/irc-logs / w3c / #testing / 2014-12-05 / end

Options:

  1. # Session Start: Fri Dec 05 00:00:00 2014
  2. # Session Ident: #testing
  3. # [00:03] * Quits: lukeis (~Adium@public.cloak) ("Leaving.")
  4. # [00:03] * Joins: lukeis (~Adium@public.cloak)
  5. # [00:16] * Quits: mdas (~mdas@public.cloak) (Client closed connection)
  6. # [00:27] * Joins: mdas (~mdas@public.cloak)
  7. # [00:29] * Joins: mdas_ (~mdas@public.cloak)
  8. # [00:29] * Quits: mdas (~mdas@public.cloak) (Client closed connection)
  9. # [00:38] * Joins: zcorpan (~zcorpan@public.cloak)
  10. # [00:45] * Quits: zcorpan (~zcorpan@public.cloak) (Ping timeout: 180 seconds)
  11. # [00:46] * Quits: mdas_ (~mdas@public.cloak) (Client closed connection)
  12. # [00:59] * Joins: bhill2_ (~bhill2@public.cloak)
  13. # [01:01] * Quits: lukeis (~Adium@public.cloak) ("Leaving.")
  14. # [01:05] * Quits: bhill2 (~bhill2@public.cloak) (Ping timeout: 180 seconds)
  15. # [01:07] * Joins: AutomatedTester (~AutomatedTester@public.cloak)
  16. # [01:12] * Joins: mdas (~mdas@public.cloak)
  17. # [01:20] * Quits: AutomatedTester (~AutomatedTester@public.cloak) (Client closed connection)
  18. # [01:20] * Quits: scott_gonzalez (~scott_gonzalez@public.cloak) (Client closed connection)
  19. # [01:21] * Joins: AutomatedTester (~AutomatedTester@public.cloak)
  20. # [01:28] * Quits: AutomatedTester (~AutomatedTester@public.cloak) (Ping timeout: 180 seconds)
  21. # [01:32] * Joins: AutomatedTester (~AutomatedTester@public.cloak)
  22. # [01:54] * Quits: AutomatedTester (~AutomatedTester@public.cloak) (Client closed connection)
  23. # [01:54] * Joins: AutomatedTester (~AutomatedTester@public.cloak)
  24. # [01:56] * Quits: bhill2_ (~bhill2@public.cloak) (Client closed connection)
  25. # [01:56] * Joins: bhill2 (~bhill2@public.cloak)
  26. # [02:01] * Quits: mdas (~mdas@public.cloak) (Client closed connection)
  27. # [02:01] * Quits: AutomatedTester (~AutomatedTester@public.cloak) (Ping timeout: 180 seconds)
  28. # [02:03] * heycam is now known as heycam|away
  29. # [02:46] * Quits: artb (~ArtB@public.cloak) ("Leaving.")
  30. # [03:14] * heycam|away is now known as heycam
  31. # [03:14] * heycam is now known as heycam|away
  32. # [03:40] * terri is now known as terri_offline
  33. # [03:49] * Joins: bhill2_ (~bhill2@public.cloak)
  34. # [03:49] <MikeSmith> are the test names the only unique identifiers we have for tests?
  35. # [03:51] <ato> I wouldn't trust test names to be unique.
  36. # [03:51] <ato> But test names + path to their containing file might be.
  37. # [03:52] <MikeSmith> well, there's a problem in that if you're storing test results and using the test names as the key, and then you notice that, e.g., the test name has a typo, so you fix that -- or change for whatever other reason -- then that's not going to match your stored key (the test name)
  38. # [03:52] <MikeSmith> that doesn't seem so good
  39. # [03:52] <ato> Yes.
  40. # [03:53] <ato> I don't know if there's any way to avoid that without marking each test with a UUID, or having some elaborate way to track test name changes.
  41. # [03:53] <MikeSmith> yeah :(
  42. # [03:54] <ato> We had the same problem with Opera's testing infrastructure which tracked test results over time in a database.
  43. # [03:54] <MikeSmith> yeah?
  44. # [03:54] <MikeSmith> doesn't seem much way around it I guess
  45. # [03:55] <ato> I mean you can invent quite elaborate technical solutions to overcome it.
  46. # [03:55] <ato> But I'm not sure it's worth it, rather than re-running the tests n number of times to build up some history.
  47. # [03:55] * Quits: bhill2 (~bhill2@public.cloak) (Ping timeout: 180 seconds)
  48. # [03:56] <ato> I guess in the case of compliance, which the wpt tests will be used for, it might be though.
  49. # [03:56] <MikeSmith> yeah
  50. # [03:56] * Quits: bhill2_ (~bhill2@public.cloak) (Ping timeout: 180 seconds)
  51. # [03:57] <MikeSmith> I think a way for test-reporting systems to deal with it is to check the file name too, as you described
  52. # [03:57] <ato> Yeah, you might have a test "A" in both files B and C.
  53. # [03:57] <ato> s/a test/tests/
  54. # [03:58] <ato> What we did with GOTH was to use test type + filename + suite name + test name.
  55. # [03:58] <ato> But we had the added complexity that some tests were sitting outside the tree, for which you needed test type.
  56. # [03:59] <MikeSmith> I was thinking the reporting system could emit a warning about any new test names added to the file since the previous run, and any test names that have dropped. Then someone watching or reading the logs of the test-report run would see that it's not a new test but instead a name change
  57. # [04:01] <ato> What would you hope to achieve by that?
  58. # [04:01] <MikeSmith> and then they'd not that they'd have to re-collect the results (if, e.g., they're collecting/aggregating results for multiple UAs)
  59. # [04:01] <MikeSmith> ato: what I just described above
  60. # [04:01] <MikeSmith> that they'd need to re-do the whole test run for all UAs
  61. # [04:01] <ato> Right, so this would essentially be useful if you wanted to do cross-browser comparisons.
  62. # [04:02] <MikeSmith> yeah
  63. # [04:02] <ato> For the regressions-over-time CI system that Mozilla uses it wouldn't really matter unless a test name change introduced a failure, in which case it would loose stability data.
  64. # [04:02] <MikeSmith> right
  65. # [04:02] <ato> (Or hope to have.)
  66. # [04:03] <ato> That would require comparing the test results to a previous report though.
  67. # [04:04] <ato> I don't know how jgraham would feel about this but I guess you could technically monitor and track renames of test() functions and filenames, since we already know what patterns we expect of filenames for them to be tests.
  68. # [04:04] <ato> (I mean it's not just test names, but file renamings also.)
  69. # [04:04] <MikeSmith> true
  70. # [04:05] <ato> If that actually proved to work one could tie it in with a git hook.
  71. # [04:06] <ato> Not sure what action would be taken when a test rename change would be detected though.
  72. # [04:06] <MikeSmith> ato: to be clear, you mean track it with wptrunner/wptupdate?
  73. # [04:06] <ato> Tracking it in wptupdate would only work if you already have a manifest, but that's actually not a bad suggestion.
  74. # [04:06] <ato> I.e. it wouldn't work with -r, if you rebuild it from scratch.
  75. # [04:07] <ato> But I was actually thinking dynamic scanning of files as they were pushed.
  76. # [04:07] <ato> Which would be much more horrible.
  77. # [04:07] <MikeSmith> ah OK
  78. # [04:07] <MikeSmith> yeah
  79. # [04:07] <ato> Your idea is better (-:
  80. # [04:07] <MikeSmith> I guess another idea would just be to have manifest.py report it
  81. # [04:08] <ato> Yeah.
  82. # [04:08] <ato> These tests went missing, these tests were added.
  83. # [04:08] <ato> I'm not sure one could make any better reporting than that without getting into computational linguistics.
  84. # [04:09] <MikeSmith> heh
  85. # [04:10] <ato> I guess it would be useful to be able to tell the cross-browser report tool that some tests changed names though, so that it would carry on the results.
  86. # [04:13] <MikeSmith> well the one that Robin created, wptreport, relies on per-browser JSON output from http://web-platform.test:8000/tools/runner/index.html in-browser runner and doesn't keep its own manifest of the tests. So currently it doesn't store anything it could use to make any comparions of the new names to old names.
  87. # [04:14] <MikeSmith> so I think for this case, we'd need to have the in-browser runner report the name differences at the time when it (re)runs manifest.py
  88. # [04:15] <MikeSmith> that wouldn't do much more than just give the person using the in-browser runner a heads-up that there's been a change
  89. # [04:15] <MikeSmith> anyway, just kind of thinking out loud
  90. # [04:16] <MikeSmith> I'll ask Robin what he thinks, once he's back
  91. # [04:26] <ato> But manifest.py only lists files right? We'd have to rely on the test results of a previous testrun and compare that against the one with the new manifest to get the full detailed list of tests changed.
  92. # [04:27] <ato> But I imagine we'd take the test diff and apply that as a filter to the JSON output of the new run.
  93. # [04:34] <MikeSmith> hmm yeah it would require something more than what I'd considered
  94. # [04:35] <MikeSmith> I think the solution for wptreport is for it to always have it just emit a message saying, "Always re-collect the results for all UAs each time you generate a new test report, because test names may have changed."
  95. # [04:36] * Joins: zcorpan (~zcorpan@public.cloak)
  96. # [04:55] * Joins: gitbot (~gitbot@public.cloak)
  97. # [04:55] -gitbot:#testing- [web-platform-tests] sideshowbarker closed pull request #1441: Ignore testharness_runner.html (master...ato/ignore_testharness_runner) https://github.com/w3c/web-platform-tests/pull/1441
  98. # [04:55] * Parts: gitbot (~gitbot@public.cloak) (gitbot)
  99. # [04:55] * Joins: gitbot (~gitbot@public.cloak)
  100. # [04:55] -gitbot:#testing- [web-platform-tests] sideshowbarker pushed 3 new commits to master: https://github.com/w3c/web-platform-tests/compare/61b58eafa239...9c0ecee186b1
  101. # [04:55] -gitbot:#testing- web-platform-tests/master 4f42370 Andreas Tolfsen: Sort .gitignore
  102. # [04:55] -gitbot:#testing- web-platform-tests/master f170f12 Andreas Tolfsen: Ignore testharness_runner.html
  103. # [04:55] -gitbot:#testing- web-platform-tests/master 9c0ecee Michael[tm] Smith: Merge pull request #1441 from andreastt/ato/ignore_testharness_runner...
  104. # [04:55] * Parts: gitbot (~gitbot@public.cloak) (gitbot)
  105. # [05:26] * Quits: zcorpan (~zcorpan@public.cloak) (Client closed connection)
  106. # [06:19] <ato> MikeSmith: I guess the big question is how often test case renames happen and how much of an issue it would be in reality.
  107. # [06:19] <ato> I imagine if one TC changes name, it likely won't affect the big picture.
  108. # [06:20] <ato> If you move a directory containing a few thousand tests, you probably know you need to regenerate the data anyway.
  109. # [06:37] * Joins: zcorpan (~zcorpan@public.cloak)
  110. # [06:38] * Joins: mdas (~mdas@public.cloak)
  111. # [06:44] * Quits: zcorpan (~zcorpan@public.cloak) (Ping timeout: 180 seconds)
  112. # [06:51] * Joins: gitbot (~gitbot@public.cloak)
  113. # [06:51] -gitbot:#testing- [web-platform-tests] zqzhang closed pull request #1414: Indexed DB: Add Event interface stub (master...inexorabletash/idb-event-stub) https://github.com/w3c/web-platform-tests/pull/1414
  114. # [06:51] * Parts: gitbot (~gitbot@public.cloak) (gitbot)
  115. # [06:51] * Joins: gitbot (~gitbot@public.cloak)
  116. # [06:51] -gitbot:#testing- [web-platform-tests] zqzhang pushed 2 new commits to master: https://github.com/w3c/web-platform-tests/compare/9c0ecee186b1...77080b62dfa7
  117. # [06:51] -gitbot:#testing- web-platform-tests/master 18a485c Joshua Bell: Add Event interface stub
  118. # [06:51] -gitbot:#testing- web-platform-tests/master 77080b6 Zhiqiang Zhang: Merge pull request #1414 from inexorabletash/inexorabletash/idb-event-stub...
  119. # [06:51] * Parts: gitbot (~gitbot@public.cloak) (gitbot)
  120. # [07:04] <jgraham> So I don't think that trying to track renaming is worthwhile
  121. # [07:04] <jgraham> It's just a special case of "what happens if you change a test in some way"
  122. # [07:05] <jgraham> And we don't have any way to track that, and I can't even think of one
  123. # [07:16] * Quits: mdas (~mdas@public.cloak) ("Leaving...")
  124. # [10:02] * Joins: Ms2ger (~Ms2ger@public.cloak)
  125. # [10:03] * Joins: Lachy (~Lachy@public.cloak)
  126. # [10:25] * Joins: zcorpan (~zcorpan@public.cloak)
  127. # [10:49] * Quits: Lachy (~Lachy@public.cloak) ("My MacBook Pro has gone to sleep. ZZZzzz…")
  128. # [10:50] * Joins: Lachy (~Lachy@public.cloak)
  129. # [11:09] * Quits: Lachy (~Lachy@public.cloak) ("My MacBook Pro has gone to sleep. ZZZzzz…")
  130. # [11:10] * Joins: Lachy (~Lachy@public.cloak)
  131. # [11:31] * Quits: Lachy (~Lachy@public.cloak) ("My MacBook Pro has gone to sleep. ZZZzzz…")
  132. # [11:33] * Joins: Lachy (~Lachy@public.cloak)
  133. # [11:33] * Joins: gitbot (~gitbot@public.cloak)
  134. # [11:33] -gitbot:#testing- [web-platform-tests] Ms2ger opened pull request #1443: Add a test for the rendering of option labels. (master...option-label) https://github.com/w3c/web-platform-tests/pull/1443
  135. # [11:33] * Parts: gitbot (~gitbot@public.cloak) (gitbot)
  136. # [11:45] * Quits: zcorpan (~zcorpan@public.cloak) (Ping timeout: 180 seconds)
  137. # [12:06] * Joins: zcorpan (~zcorpan@public.cloak)
  138. # [12:10] * Joins: gitbot (~gitbot@public.cloak)
  139. # [12:10] -gitbot:#testing- [web-platform-tests] zcorpan closed pull request #1443: Add a test for the rendering of option labels. (master...option-label) https://github.com/w3c/web-platform-tests/pull/1443
  140. # [12:10] * Parts: gitbot (~gitbot@public.cloak) (gitbot)
  141. # [12:10] * Joins: gitbot (~gitbot@public.cloak)
  142. # [12:10] -gitbot:#testing- [web-platform-tests] zcorpan pushed 2 new commits to master: https://github.com/w3c/web-platform-tests/compare/77080b62dfa7...b76cb1543ec6
  143. # [12:10] -gitbot:#testing- web-platform-tests/master 45e3eeb Ms2ger: Add a test for the rendering of option labels....
  144. # [12:10] -gitbot:#testing- web-platform-tests/master b76cb15 Simon Pieters: Merge pull request #1443 from Ms2ger/option-label...
  145. # [12:10] * Parts: gitbot (~gitbot@public.cloak) (gitbot)
  146. # [12:31] * Joins: abarsto (~abarsto@public.cloak)
  147. # [12:31] * abarsto is now known as ArtB
  148. # [12:35] * Quits: Ms2ger (~Ms2ger@public.cloak) (Ping timeout: 180 seconds)
  149. # [12:39] * Joins: zcorpan_ (~zcorpan@public.cloak)
  150. # [12:39] * Quits: zcorpan (~zcorpan@public.cloak) (Client closed connection)
  151. # [13:03] * Quits: zcorpan_ (~zcorpan@public.cloak) (Ping timeout: 180 seconds)
  152. # [13:25] * Joins: zcorpan (~zcorpan@public.cloak)
  153. # [14:03] * Joins: Ms2ger (~Ms2ger@public.cloak)
  154. # [14:11] * Joins: AutomatedTester (~AutomatedTester@public.cloak)
  155. # [14:18] * Quits: Ms2ger (~Ms2ger@public.cloak) (Ping timeout: 180 seconds)
  156. # [14:29] * Joins: plh (plehegar@public.cloak)
  157. # [15:53] * Quits: zcorpan (~zcorpan@public.cloak) (Client closed connection)
  158. # [16:19] * Quits: shepazu (schepers@public.cloak)
  159. # [16:21] * Joins: shepazu (schepers@public.cloak)
  160. # [16:24] * Quits: shepazu (schepers@public.cloak) ("is sleepy")
  161. # [16:25] * Joins: Ms2ger (~Ms2ger@public.cloak)
  162. # [16:54] * Joins: zcorpan (~zcorpan@public.cloak)
  163. # [17:01] * Quits: zcorpan (~zcorpan@public.cloak) (Ping timeout: 180 seconds)
  164. # [17:17] * Joins: scott_gonzalez (~scott_gonzalez@public.cloak)
  165. # [17:30] * Joins: gitbot (~gitbot@public.cloak)
  166. # [17:30] -gitbot:#testing- [web-platform-tests] hallvors pushed 1 new commit to hallvors/clipboard-api-2: https://github.com/w3c/web-platform-tests/commit/cc3f9546220484b8c1d80fac7bee729065a8f000
  167. # [17:30] -gitbot:#testing- web-platform-tests/hallvors/clipboard-api-2 cc3f954 Hallvord R. M. Steen: giving test files more meaningful names, avoid numbers that change every time the test suite is built
  168. # [17:30] * Parts: gitbot (~gitbot@public.cloak) (gitbot)
  169. # [17:32] * Quits: AutomatedTester (~AutomatedTester@public.cloak) (Client closed connection)
  170. # [17:34] * Quits: ArtB (~abarsto@public.cloak) ("Leaving.")
  171. # [17:47] * Quits: Lachy (~Lachy@public.cloak) ("My MacBook Pro has gone to sleep. ZZZzzz…")
  172. # [17:51] * Quits: plh (plehegar@public.cloak) ("Leaving")
  173. # [17:51] * Joins: plh (plehegar@public.cloak)
  174. # [17:56] * Joins: artb (~ArtB@public.cloak)
  175. # [18:09] * Joins: bhill2 (~bhill2@public.cloak)
  176. # [18:09] * Quits: bhill2 (~bhill2@public.cloak) (Client closed connection)
  177. # [18:09] * Joins: bhill2 (~bhill2@public.cloak)
  178. # [18:10] * Joins: lukeis (~Adium@public.cloak)
  179. # [18:42] * Joins: zcorpan (~zcorpan@public.cloak)
  180. # [18:49] * Quits: zcorpan (~zcorpan@public.cloak) (Ping timeout: 180 seconds)
  181. # [18:53] * Joins: mdas (~mdas@public.cloak)
  182. # [19:14] * heycam|away is now known as heycam
  183. # [19:19] * Joins: AutomatedTester (~AutomatedTester@public.cloak)
  184. # [19:20] * Quits: plh (plehegar@public.cloak) (Client closed connection)
  185. # [19:21] * Joins: plh (plehegar@public.cloak)
  186. # [19:53] * Joins: zcorpan (~zcorpan@public.cloak)
  187. # [19:54] * Quits: AutomatedTester (~AutomatedTester@public.cloak) (Client closed connection)
  188. # [19:55] * Joins: AutomatedTester (~AutomatedTester@public.cloak)
  189. # [19:56] <ato> jgraham: Can you do the last merge commit in https://critic.hoppipolla.co.uk/r/2949 so I can clean it up and merge it?
  190. # [20:00] * Quits: zcorpan (~zcorpan@public.cloak) (Ping timeout: 180 seconds)
  191. # [20:02] * Quits: AutomatedTester (~AutomatedTester@public.cloak) (Ping timeout: 180 seconds)
  192. # [20:03] * heycam is now known as heycam|away
  193. # [20:04] * Joins: AutomatedTester (~AutomatedTester@public.cloak)
  194. # [20:12] * heycam|away is now known as heycam
  195. # [20:24] <jgraham> ato: Done
  196. # [20:36] * Joins: mdas_ (~mdas@public.cloak)
  197. # [20:36] * Quits: mdas (~mdas@public.cloak) (Client closed connection)
  198. # [20:44] * heycam is now known as heycam|away
  199. # [20:45] * heycam|away is now known as heycam
  200. # [20:47] * Joins: Lachy (~Lachy@public.cloak)
  201. # [20:56] * Quits: AutomatedTester (~AutomatedTester@public.cloak) (Client closed connection)
  202. # [20:56] * Joins: AutomatedTester (~AutomatedTester@public.cloak)
  203. # [21:03] * Quits: AutomatedTester (~AutomatedTester@public.cloak) (Ping timeout: 180 seconds)
  204. # [21:13] * terri_offline is now known as terri
  205. # [21:15] * Joins: AutomatedTester (~AutomatedTester@public.cloak)
  206. # [21:26] * Quits: plh (plehegar@public.cloak) ("Leaving")
  207. # [21:26] * Joins: plh (plehegar@public.cloak)
  208. # [21:38] * Quits: mdas_ (~mdas@public.cloak) (Client closed connection)
  209. # [21:48] * Joins: gitbot (~gitbot@public.cloak)
  210. # [21:48] -gitbot:#testing- [wptrunner] andreastt pushed 1 new commit to ato/encapsulate_webdriver: https://github.com/w3c/wptrunner/commit/682290cc5104df4afb887d0b8417ae27965a5c8e
  211. # [21:48] -gitbot:#testing- wptrunner/ato/encapsulate_webdriver 682290c Andreas Tolfsen: fixup! Add support for different WebDriver servers
  212. # [21:48] * Parts: gitbot (~gitbot@public.cloak) (gitbot)
  213. # [21:52] * Joins: mdas (~mdas@public.cloak)
  214. # [22:00] * Quits: AutomatedTester (~AutomatedTester@public.cloak) (Client closed connection)
  215. # [22:01] * Joins: AutomatedTester (~AutomatedTester@public.cloak)
  216. # [22:01] * Quits: AutomatedTester (~AutomatedTester@public.cloak) (Client closed connection)
  217. # [22:02] * Joins: AutomatedTester (~AutomatedTester@public.cloak)
  218. # [22:09] * Quits: AutomatedTester (~AutomatedTester@public.cloak) (Ping timeout: 180 seconds)
  219. # [22:09] * heycam is now known as heycam|away
  220. # [22:10] * Joins: AutomatedTester (~AutomatedTester@public.cloak)
  221. # [22:13] * heycam|away is now known as heycam
  222. # [22:13] * Quits: plh (plehegar@public.cloak) ("Leaving")
  223. # [22:36] * Joins: plh (plehegar@public.cloak)
  224. # [22:40] * Quits: Ms2ger (~Ms2ger@public.cloak) ("nn")
  225. # [22:41] * Joins: Ms2ger (~Ms2ger@public.cloak)
  226. # [22:41] * Quits: Ms2ger (~Ms2ger@public.cloak) (Client closed connection)
  227. # [22:43] * Quits: plh (plehegar@public.cloak) ("Leaving")
  228. # [23:12] * heycam is now known as heycam|away
  229. # [23:18] * Quits: mdas (~mdas@public.cloak) (Client closed connection)
  230. # [23:30] * Joins: mdas (~mdas@public.cloak)
  231. # [23:41] * Quits: mdas (~mdas@public.cloak) (Client closed connection)
  232. # Session Close: Sat Dec 06 00:00:00 2014

The end :)