/irc-logs / w3c / #testing / 2013-08-22 / end

Options:

  1. # Session Start: Thu Aug 22 00:00:00 2013
  2. # Session Ident: #testing
  3. # [00:00] * Quits: rhauck (~Adium@public.cloak) (Ping timeout: 180 seconds)
  4. # [00:15] * Quits: zcorpan (~zcorpan@public.cloak) (Client closed connection)
  5. # [00:16] * Joins: zcorpan (~zcorpan@public.cloak)
  6. # [00:23] * Quits: zcorpan (~zcorpan@public.cloak) (Ping timeout: 180 seconds)
  7. # [00:25] * Quits: tobie (tobie@public.cloak) (Client closed connection)
  8. # [00:26] * Joins: tobie (tobie@public.cloak)
  9. # [00:46] * Joins: zcorpan (~zcorpan@public.cloak)
  10. # [00:58] * Quits: rhauck1 (~Adium@public.cloak) ("Leaving.")
  11. # [01:01] * Joins: rhauck (~Adium@public.cloak)
  12. # [01:12] * Quits: zcorpan (~zcorpan@public.cloak) (Ping timeout: 180 seconds)
  13. # [01:26] * heycam|away is now known as heycam
  14. # [01:37] * Joins: zcorpan (~zcorpan@public.cloak)
  15. # [01:47] * Quits: kershaw (~kkershaw@public.cloak) (Ping timeout: 180 seconds)
  16. # [02:08] * Quits: tobie (tobie@public.cloak)
  17. # [02:20] * Quits: zcorpan (~zcorpan@public.cloak) (Ping timeout: 180 seconds)
  18. # [02:30] * Quits: jhammel (~jhammel@public.cloak) ("leaving")
  19. # [02:44] * Joins: zcorpan (~zcorpan@public.cloak)
  20. # [02:54] * Joins: gitbot (~gitbot@public.cloak)
  21. # [02:54] -gitbot:#testing- [web-platform-tests] zqzhang synchronize pull request #296: Added idlharness test for animation-timing (master...submission/zqzhang/animation-timing-idlharness) https://github.com/w3c/web-platform-tests/pull/296
  22. # [02:54] * Parts: gitbot (~gitbot@public.cloak) (gitbot)
  23. # [03:35] * Joins: gitbot (~gitbot@public.cloak)
  24. # [03:35] -gitbot:#testing- [web-platform-tests] ibelem synchronize pull request #280: draggable_attribute, dropzone_attribute and dragevent (master...submission/ibelem/draganddrop) https://github.com/w3c/web-platform-tests/pull/280
  25. # [03:35] * Parts: gitbot (~gitbot@public.cloak) (gitbot)
  26. # [04:23] * Quits: zcorpan (~zcorpan@public.cloak) (Ping timeout: 180 seconds)
  27. # [04:33] * heycam is now known as heycam|away
  28. # [04:54] * Quits: rhauck (~Adium@public.cloak) ("Leaving.")
  29. # [04:55] * Joins: rhauck (~Adium@public.cloak)
  30. # [05:02] * Quits: rhauck (~Adium@public.cloak) (Ping timeout: 180 seconds)
  31. # [05:11] * Joins: kershaw (~kkershaw@public.cloak)
  32. # [05:11] * heycam|away is now known as heycam
  33. # [05:29] * Joins: MikeSmith_ (~MikeSmith@public.cloak)
  34. # [05:30] * Quits: MikeSmith_ (~MikeSmith@public.cloak) ("Make the road by walking. (B-side: Tired of Fighting)")
  35. # [07:15] * Quits: kershaw (~kkershaw@public.cloak) (Ping timeout: 180 seconds)
  36. # [07:43] * Joins: zcorpan (~zcorpan@public.cloak)
  37. # [07:44] * Quits: zcorpan (~zcorpan@public.cloak) (Client closed connection)
  38. # [07:44] * Joins: zcorpan (~zcorpan@public.cloak)
  39. # [08:08] * Quits: zcorpan (~zcorpan@public.cloak) (Client closed connection)
  40. # [08:08] * Joins: zcorpan (~zcorpan@public.cloak)
  41. # [08:15] * Quits: zcorpan (~zcorpan@public.cloak) (Ping timeout: 180 seconds)
  42. # [08:19] * Joins: Ms2ger (~Ms2ger@public.cloak)
  43. # [08:21] * Joins: tobie (tobie@public.cloak)
  44. # [09:09] * Joins: gitbot (~gitbot@public.cloak)
  45. # [09:09] -gitbot:#testing- [web-platform-tests] ibelem synchronize pull request #280: draggable_attribute, dropzone_attribute and dragevent (master...submission/ibelem/draganddrop) https://github.com/w3c/web-platform-tests/pull/280
  46. # [09:09] * Parts: gitbot (~gitbot@public.cloak) (gitbot)
  47. # [09:31] * Joins: gitbot (~gitbot@public.cloak)
  48. # [09:31] -gitbot:#testing- [web-platform-tests] ibelem synchronize pull request #280: draggable_attribute, dropzone_attribute and dragevent (master...submission/ibelem/draganddrop) https://github.com/w3c/web-platform-tests/pull/280
  49. # [09:31] * Parts: gitbot (~gitbot@public.cloak) (gitbot)
  50. # [09:36] * Joins: gitbot (~gitbot@public.cloak)
  51. # [09:36] -gitbot:#testing- [web-platform-tests] ibelem synchronize pull request #280: draggable_attribute, dropzone_attribute and dragevent (master...submission/ibelem/draganddrop) https://github.com/w3c/web-platform-tests/pull/280
  52. # [09:36] * Parts: gitbot (~gitbot@public.cloak) (gitbot)
  53. # [09:48] * Quits: Lachy (~Lachy@public.cloak) ("Computer has gone to sleep.")
  54. # [09:52] * Joins: Lachy (~Lachy@public.cloak)
  55. # [09:53] * Quits: Lachy (~Lachy@public.cloak) ("Computer has gone to sleep.")
  56. # [09:56] * Joins: zcorpan (~zcorpan@public.cloak)
  57. # [10:22] * Joins: Lachy (~Lachy@public.cloak)
  58. # [10:49] * heycam is now known as heycam|away
  59. # [10:58] * Joins: AutomatedTester (~AutomatedTester@public.cloak)
  60. # [13:40] * Quits: zcorpan (~zcorpan@public.cloak) (Client closed connection)
  61. # [13:40] * Joins: zcorpan (~zcorpan@public.cloak)
  62. # [13:41] * Joins: Automate_ (~AutomatedTester@public.cloak)
  63. # [13:47] * Quits: AutomatedTester (~AutomatedTester@public.cloak) (Ping timeout: 180 seconds)
  64. # [13:47] * Quits: zcorpan (~zcorpan@public.cloak) (Ping timeout: 180 seconds)
  65. # [13:48] * Quits: Automate_ (~AutomatedTester@public.cloak) (Ping timeout: 180 seconds)
  66. # [14:00] * Joins: darobin (rberjon@public.cloak)
  67. # [14:12] * Joins: zcorpan (~zcorpan@public.cloak)
  68. # [14:40] * Joins: dom (dom@public.cloak)
  69. # [14:58] * Quits: zcorpan (~zcorpan@public.cloak) (Ping timeout: 180 seconds)
  70. # [15:22] * Joins: zcorpan (~zcorpan@public.cloak)
  71. # [15:28] * Quits: zcorpan (~zcorpan@public.cloak) (Client closed connection)
  72. # [15:31] * Joins: zcorpan (~zcorpan@public.cloak)
  73. # [15:41] * Quits: zcorpan (~zcorpan@public.cloak) (Client closed connection)
  74. # [15:41] * Joins: zcorpan (~zcorpan@public.cloak)
  75. # [15:48] * Quits: zcorpan (~zcorpan@public.cloak) (Ping timeout: 180 seconds)
  76. # [16:10] * Quits: dom (dom@public.cloak) ("")
  77. # [16:12] * Joins: zcorpan (~zcorpan@public.cloak)
  78. # [16:59] * Quits: zcorpan (~zcorpan@public.cloak) (Client closed connection)
  79. # [17:00] * Joins: zcorpan (~zcorpan@public.cloak)
  80. # [17:01] * Joins: zcorpan_ (~zcorpan@public.cloak)
  81. # [17:01] * Quits: zcorpan (~zcorpan@public.cloak) (Client closed connection)
  82. # [17:04] * Joins: kershaw (~kkershaw@public.cloak)
  83. # [17:08] * Quits: zcorpan_ (~zcorpan@public.cloak) (Ping timeout: 180 seconds)
  84. # [17:19] <jgraham> darobin: Really? You think that it would work well to require authors to copy an existing file every time they want to reuse a reference is a win compared to just pointing at an existing file?
  85. # [17:19] <jgraham> Uh, that sentence got lost halfway through
  86. # [17:19] <jgraham> But you see what I mean
  87. # [17:20] * darobin looks at timer — and it's a new record for jgraham jumping from email to IRC in anger!!!! The crowd goes wild.
  88. # [17:20] <Ms2ger> Hey darobin!
  89. # [17:20] <darobin> hey Ms2ger!
  90. # [17:20] <jgraham> Also, having to read and hash every file would be much slower than just comparing urls
  91. # [17:20] * jgraham isn't angry :p
  92. # [17:21] <darobin> jgraham: to be honest, I don't care about that specific issue :)
  93. # [17:21] <Ms2ger> darobin, have you had time to look at the canvas code?
  94. # [17:21] <darobin> we could use naming conventions that identify resources as references without them having the same name as the file they're references for
  95. # [17:21] <darobin> Ms2ger: I just came back from vacation
  96. # [17:21] <darobin> (again)
  97. # [17:21] <darobin> well, from a long weekend
  98. # [17:21] <darobin> so no
  99. # [17:21] <Ms2ger> You must be French :)
  100. # [17:21] <darobin> but I plan to, unless beaten to it
  101. # [17:21] <darobin> Ms2ger: you guess well :)
  102. # [17:25] <jgraham> darobin: That doesn't solve anything really. You still need metadata to say which ref goes with which file, which currently seems to be <meta> elements in-file. Which is too slow to read all the time. Therefore you either need a build step to generate manifests or checked-in manifests
  103. # [17:25] <darobin> yeah
  104. # [17:25] <darobin> go build step!
  105. # [17:25] <Ms2ger> Boo build step!
  106. # [17:25] <darobin> it solves the "identifying which files contain tests" problem
  107. # [17:25] <darobin> which is the one I care about more
  108. # [17:25] <darobin> linking to ref tests, well, that can use a build step
  109. # [17:25] <jgraham> I think my feeling about build step is "meh". It doesn't solve the "I want to specify a URL as a test" problem
  110. # [17:25] <darobin> I just don't want to have to see conflicts from manifests ever again
  111. # [17:26] <jgraham> But Mozilla is going to need some hideous thing to work out which tests are expected to fail
  112. # [17:26] <darobin> for the subset of tests where the mapping from URL to file is not obvious, you can specify that
  113. # [17:26] <darobin> oh this will require hideous stuff no matter what :)
  114. # [17:26] <darobin> we're just arguing about which hideous colour to paint the horror
  115. # [17:28] <jgraham> I guess I could live with file[name] based metadata for most things and explicit MANIFEST files to override the extracted data where required.
  116. # [17:28] <jgraham> And a checked in tool that builds a full manifest so that everyone agrees on what the actual data is
  117. # [17:28] <darobin> jgraham: to be frank about the whole thing, given that you seem to have the bandwidth to put in the work I reckon you get to pick the solution anyway
  118. # [17:29] <jgraham> darobin: There is some truth in that :)
  119. # [17:29] * Ms2ger pushes jgraham towards manifests
  120. # [17:29] <darobin> jgraham: yeah, I think that if we can have vernacular filename based information and only resort to manifests when needed I'd be happy, and we'd all be happier in the long run
  121. # [17:30] <darobin> manifests just don't play well with branches and stuff
  122. # [17:30] * Joins: zcorpan (~zcorpan@public.cloak)
  123. # [17:30] <jgraham> I have to say that I can foresee manifests causing merge issues
  124. # [17:30] * Quits: Lachy (~Lachy@public.cloak) ("Computer has gone to sleep.")
  125. # [17:30] <darobin> I wonder if it would also be possible to use a file format slightly less ad hoc than the one in manifestdestiny :)
  126. # [17:31] <jgraham> That aren't hard to solve, but require people to understand how to resolve merge conflicts
  127. # [17:31] <Ms2ger> Dunno, you only get merge issues if you have a small manifest and a lot of changes to it
  128. # [17:31] <darobin> jgraham: they will cause merging issues, but perhaps more annoying is the problem of how they will likely become corrupted when a frustrated merger goes through the manifest conflicts too fast
  129. # [17:31] <jgraham> Manifests out to merge very well
  130. # [17:31] <jgraham> *ought
  131. # [17:32] <darobin> that's not my experience :)
  132. # [17:32] <Ms2ger> Having a check that all files in the repo are mentioned in a manifest should be sufficient, surely
  133. # [17:32] <jgraham> If they are just lists the merge conflict resolution is almost always "take the change from both branches"
  134. # [17:34] <darobin> mmmm
  135. # [17:34] <jgraham> But of course it is always easy to make a mistake when merging
  136. # [17:34] <darobin> so maybe solve that with CI?
  137. # [17:34] * Ms2ger already wrote such a script once
  138. # [17:36] <jgraham> Ms2ger: So what's the argument in favour of explicit manifests vs some sort of generated manifest?
  139. # [17:37] <Ms2ger> 1. bad experiences with forcing a build step in the css tests
  140. # [17:37] <jgraham> I seem to recall that was bad too, but I don't remember the specific badness
  141. # [17:37] <Ms2ger> 2. having manifests that are only used for some tests seems unnecessarily confusing
  142. # [17:39] <jgraham> Re: 2. I think it's OK to have features that are for advanced users only. People like Aryeh who are writing complex, long-running, tests can deal with the confusion of having to edit a file like override.manifest
  143. # [17:40] <darobin> yeah I agree
  144. # [17:41] <Ms2ger> Also, istr that for reftests in particular, the in-test approach is a lot more verbose than a typical manifest
  145. # [17:42] <jgraham> It is
  146. # [17:42] <darobin> do you need anything more in-test than a pointer to the ref?
  147. # [17:42] <Ms2ger> <link rel="reference" href="...
  148. # [17:42] <jgraham> Well accorind to the CSSWG you should spend about half an hour writing metadata
  149. # [17:43] <jgraham> per test
  150. # [17:45] * Ms2ger wonders how to distinguish partially-manual and fully-automatic testharness tests
  151. # [17:45] <jgraham> (link rel=authou, rel=flags rel=assert, rel=reviewer, rel=match|mismatch
  152. # [17:45] <jgraham> )
  153. # [17:45] <jgraham> *author
  154. # [17:46] <jgraham> (I find rel=reviewer particularly funny. A test gets reviewed OK and then you have to change it and hope that doesn't invalidate the test)
  155. # [17:47] <jgraham> Ms2ger: All tests that aren't sully automatic are manual, I guess
  156. # [17:47] <jgraham> *fully
  157. # [17:47] <Ms2ger> Well, yes
  158. # [17:47] <Ms2ger> How do you tell?
  159. # [17:47] <jgraham> Well there are two conventions I know of
  160. # [17:47] <jgraham> foo-manual.html
  161. # [17:47] <jgraham> and
  162. # [17:47] <jgraham> non-automated/foo.html
  163. # [17:48] <Ms2ger> Neither of those is used in wpt, though
  164. # [17:48] <jgraham> I thought they were
  165. # [17:48] <jgraham> (not consistently)
  166. # [17:49] <jgraham> /workers/non-automated/
  167. # [17:50] <jgraham> jgraham@luna:~/develop/web-platform-tests$ find . -iname *MANUAL* | wc -l
  168. # [17:50] <jgraham> 22
  169. # [17:50] * Joins: rhauck (~Adium@public.cloak)
  170. # [17:52] <Ms2ger> Also -interactive, apparently
  171. # [17:53] <Ms2ger> A bunch of the foreign content parsing tests from MS are unmarked
  172. # [17:54] <Ms2ger> And a few nested-document-write ones
  173. # [17:54] <jgraham> Yeah, not surprising given that there is no strong convention
  174. # [17:54] <jgraham> Why are those manual tests?
  175. # [17:54] <jgraham> I guess the no-manifest thing doesn't really solve the timeout problem
  176. # [17:55] <jgraham> Quite a few tests need unusually long timeouts
  177. # [17:55] <jgraham> I'm not super-excited about having to run the files to find this out.
  178. # [17:58] * Joins: zcorpan_ (~zcorpan@public.cloak)
  179. # [17:58] * Quits: zcorpan (~zcorpan@public.cloak) (Client closed connection)
  180. # [17:59] * Ms2ger is not super-excited about running files in general
  181. # [17:59] <jgraham> I quite like running them to get the test results out
  182. # [17:59] <jgraham> It always struck me as useful
  183. # [18:00] <Ms2ger> Meh
  184. # [18:00] * Ms2ger gets dinner
  185. # [18:00] <darobin> well there's a file naming convention for everything
  186. # [18:01] <darobin> you could have foo-seems-to-go-on-forever.html
  187. # [18:01] * Joins: Lachy (~Lachy@public.cloak)
  188. # [18:01] <darobin> for those, bump the timeout to, I don't know 300s
  189. # [18:02] <darobin> in fact, thinking about it, I'm realising that the ability to specify a numeric time out is undesirable
  190. # [18:02] <darobin> you only want two values: "regular" and "might take a fucking long time"
  191. # [18:03] <darobin> the reason being that you don't know what context your tests are being run in, and so don't know what's reasonable and long
  192. # [18:03] <tobie> darobin: nah, you want PER DEVICE numeric values
  193. # [18:03] <jgraham> darobin: Or climb down from my anti-meta stance and say that rather than doing setup({timeout:10000}) do <meta name=timeout value=10> and have the setup function read from that
  194. # [18:03] * darobin kicks tobie
  195. # [18:03] <jgraham> And if you can't add <meta> use the manifest
  196. # [18:04] <darobin> jgraham: honestly, I still don't think that's all that useful due to it being numeric
  197. # [18:04] <jgraham> darobin: No, the person running the tests applies a device-specific multiplier
  198. # [18:04] <darobin> mmmmm, maybe
  199. # [18:04] <jgraham> That was what we did at Opera
  200. # [18:04] <darobin> that works?
  201. # [18:04] <jgraham> It wasn't perfect but it was OK
  202. # [18:04] <jgraham> I mean
  203. # [18:04] <tobie> what's the use case for long timeouts, btw?
  204. # [18:05] <jgraham> You don't want to wait 300s for a test that is supposed to take 11s just because the default is 10
  205. # [18:05] <darobin> tobie: tests that do loads of stuff
  206. # [18:05] <gsnedders> tobie: Imagine a test that loads a file over XHR, using a script such that only 10 bytes are sent per second.
  207. # [18:05] <darobin> jgraham: right, but then again if you work from a multiplier it means that everyone has to enter based on the same expectations
  208. # [18:06] <jgraham> Yes
  209. # [18:06] <gsnedders> The problem with multipliers is not everything scales equally. IO (for cache behaviour) can be radically different to CPU speed, for example.
  210. # [18:06] <darobin> jgraham: say I checked my test on my old iphone 3g because I'm that kind of guy, and updated the timeout accordingly... you're going to have a bad time multiplying that
  211. # [18:06] <tobie> isn't there a way to ping the framework at regular intervals in such cases?
  212. # [18:07] <gsnedders> tobie: Okay then, sync XHR.
  213. # [18:07] <jgraham> tobie: that sounds way too invasive
  214. # [18:07] <gsnedders> tobie: Testing that it doesn't give up after 20s.
  215. # [18:08] <tobie> gsnedders: that's really QoI tests
  216. # [18:08] <jgraham> Not at all
  217. # [18:09] <tobie> jgraham: enlighten me
  218. # [18:10] * Quits: rhauck (~Adium@public.cloak) ("Leaving.")
  219. # [18:10] <jgraham> well, everything is testing the QoI to some extent. So there are two ways to look at this: a) apart from the hardware limitations clause there is no provision in the standard for giving up on operations so it is a conformance problem if you do so and b) we actually don't care about QoI vs conformance; we care about interop and should test all the things we know affect interop
  220. # [18:11] <jgraham> rather than arbitarily rejecting tests (that might well be based on actual browser bugs)
  221. # [18:11] <Ms2ger> Fwiw, mochitests have a requestLongerTimeout(factor) that just multiplies the default timeout
  222. # [18:12] <tobie> jgraham: thanks
  223. # [18:13] <jgraham> Ms2ger: It isn't clear to me that's materially different to a ms-based timeout and a per-platform multiplier
  224. # [18:14] <Ms2ger> Maybe not
  225. # [18:14] <Ms2ger> But I guess that means that a 10s timeout doesn't mean you time out after 10s
  226. # [18:17] * Quits: zcorpan_ (~zcorpan@public.cloak) (Client closed connection)
  227. # [18:18] * Joins: zcorpan (~zcorpan@public.cloak)
  228. # [18:25] * Quits: zcorpan (~zcorpan@public.cloak) (Ping timeout: 180 seconds)
  229. # [18:30] * Joins: zcorpan (~zcorpan@public.cloak)
  230. # [18:45] * Joins: rhauck (~Adium@public.cloak)
  231. # [19:00] * Joins: [1]kershaw (~kkershaw@public.cloak)
  232. # [19:05] * Quits: kershaw (~kkershaw@public.cloak) (Ping timeout: 180 seconds)
  233. # [19:05] * [1]kershaw is now known as kershaw
  234. # [19:05] * Quits: darobin (rberjon@public.cloak) (Client closed connection)
  235. # [19:06] * Joins: jhammel (~jhammel@public.cloak)
  236. # [20:38] * Joins: AutomatedTester (~AutomatedTester@public.cloak)
  237. # [20:57] * Quits: Ms2ger (~Ms2ger@public.cloak) ("nn")
  238. # [21:20] * Joins: jimevans (~jimevans@public.cloak)
  239. # [21:42] * Quits: jimevans (~jimevans@public.cloak) ("Colloquy for iPhone - http://colloquy.mobi")
  240. # [22:22] * heycam|away is now known as heycam
  241. # [22:23] * Quits: zcorpan (~zcorpan@public.cloak) (Client closed connection)
  242. # [22:24] * Joins: zcorpan (~zcorpan@public.cloak)
  243. # [22:31] * Quits: zcorpan (~zcorpan@public.cloak) (Ping timeout: 180 seconds)
  244. # [22:54] * Joins: zcorpan (~zcorpan@public.cloak)
  245. # [23:31] * Quits: zcorpan (~zcorpan@public.cloak) (Ping timeout: 180 seconds)
  246. # [23:40] * Quits: shepazu (schepers@public.cloak) ("is sleepy")
  247. # [23:55] * Joins: zcorpan (~zcorpan@public.cloak)
  248. # Session Close: Fri Aug 23 00:00:00 2013

The end :)