/irc-logs / w3c / #testing / 2013-03-21 / end

Options:

  1. # Session Start: Thu Mar 21 00:00:00 2013
  2. # Session Ident: #testing
  3. # [00:07] * Quits: glenn (~gadams@public.cloak) (Client closed connection)
  4. # [00:31] * Joins: glenn (~gadams@public.cloak)
  5. # [03:38] * Joins: glenn_ (~gadams@public.cloak)
  6. # [03:38] * Quits: glenn (~gadams@public.cloak) (Client closed connection)
  7. # [03:49] * Quits: glenn_ (~gadams@public.cloak) (Client closed connection)
  8. # [04:47] * Joins: glenn (~gadams@public.cloak)
  9. # [06:49] * Quits: glenn (~gadams@public.cloak) (Client closed connection)
  10. # [07:54] * Joins: tobie (tobie@public.cloak)
  11. # [08:00] * Joins: glenn (~gadams@public.cloak)
  12. # [08:04] * Quits: glenn (~gadams@public.cloak) (Ping timeout: 60 seconds)
  13. # [08:29] * Quits: tobie (tobie@public.cloak)
  14. # [08:50] * Joins: tobie (tobie@public.cloak)
  15. # [08:52] * Quits: tobie (tobie@public.cloak)
  16. # [09:24] * Joins: darobin (rberjon@public.cloak)
  17. # [09:44] * Joins: zcorpan (~zcorpan@public.cloak)
  18. # [10:10] * Quits: zcorpan (~zcorpan@public.cloak) (Client closed connection)
  19. # [10:11] * Joins: zcorpan (~zcorpan@public.cloak)
  20. # [10:14] * Joins: AutomatedTester (~AutomatedTester@public.cloak)
  21. # [10:43] * Quits: AutomatedTester (~AutomatedTester@public.cloak) (Client closed connection)
  22. # [11:25] * Joins: abarsto (~abarsto@public.cloak)
  23. # [11:25] * abarsto is now known as ArtB
  24. # [12:51] * Joins: darobin_ (rberjon@public.cloak)
  25. # [12:51] * Quits: darobin (rberjon@public.cloak) (Client closed connection)
  26. # [12:58] * Joins: plh (plehegar@public.cloak)
  27. # [13:01] * Joins: tobie (tobie@public.cloak)
  28. # [13:01] * darobin_ is now known as darobin
  29. # [13:33] * Quits: shepazu (schepers@public.cloak) ("is sleepy")
  30. # [15:29] * Joins: shepazu (schepers@public.cloak)
  31. # [15:55] * Joins: AutomatedTester (~AutomatedTester@public.cloak)
  32. # [15:59] * Quits: AutomatedTester (~AutomatedTester@public.cloak) (Ping timeout: 60 seconds)
  33. # [16:34] * Joins: AutomatedTester (~AutomatedTester@public.cloak)
  34. # [17:13] * Joins: jhammel (~jhammel@public.cloak)
  35. # [17:15] * Joins: glenn (~gadams@public.cloak)
  36. # [17:19] * Quits: zcorpan (~zcorpan@public.cloak) (Client closed connection)
  37. # [17:53] * Quits: stearns (~anonymous@public.cloak) (stearns)
  38. # [17:56] * Joins: stearns (~anonymous@public.cloak)
  39. # [17:57] * Quits: shepazu (schepers@public.cloak) (Ping timeout: 60 seconds)
  40. # [18:00] * Joins: shepazu (schepers@public.cloak)
  41. # [18:02] * Joins: Ms2ger (~Ms2ger@public.cloak)
  42. # [18:03] * Quits: AutomatedTester (~AutomatedTester@public.cloak) (Client closed connection)
  43. # [18:06] * Quits: darobin (rberjon@public.cloak) (Client closed connection)
  44. # [18:06] * Joins: darobin (rberjon@public.cloak)
  45. # [18:30] * Quits: darobin (rberjon@public.cloak) (Client closed connection)
  46. # [18:53] * Joins: zcorpan (~zcorpan@public.cloak)
  47. # [18:55] * Quits: glenn (~gadams@public.cloak) (Client closed connection)
  48. # [19:02] * Quits: shepazu (schepers@public.cloak) (Ping timeout: 60 seconds)
  49. # [19:02] * Quits: zcorpan (~zcorpan@public.cloak) (Client closed connection)
  50. # [19:03] * Joins: zcorpan (~zcorpan@public.cloak)
  51. # [19:07] * Quits: zcorpan (~zcorpan@public.cloak) (Ping timeout: 60 seconds)
  52. # [19:13] * Joins: AutomatedTester (~AutomatedTester@public.cloak)
  53. # [19:17] * Quits: AutomatedTester (~AutomatedTester@public.cloak) (Ping timeout: 60 seconds)
  54. # [19:38] * Joins: zcorpan (~zcorpan@public.cloak)
  55. # [19:54] <ArtB> tobie - re your "Review of tests upstream …" thread ...
  56. # [19:55] <ArtB> in the GH world, does a spec still have directories for submissions?
  57. # [19:56] <ArtB> e.g. webstorage/submissions/Tobie/foo.html
  58. # [19:58] <Ms2ger> No
  59. # [20:03] <ArtB> oh, that's right the "submissions" are in someone else's GH home?
  60. # [20:05] * Joins: glenn (~gadams@public.cloak)
  61. # [20:05] <tobie> ArtB: pretty much, yes.
  62. # [20:09] <ArtB> so, in the case of the upstream streamlining, you propose those tests be automatically approved if the upstream contributor can show the tests have had "adequate review"?
  63. # [20:09] * Quits: glenn (~gadams@public.cloak) (Client closed connection)
  64. # [20:09] * Joins: glenn (~gadams@public.cloak)
  65. # [20:16] <jgraham> ArtB: Yes, that's the proposal AIUI
  66. # [20:19] <ArtB> jgraham, but that doesn't replace the WG's responsibility to approve tests used to determine CR exit criteria. Correct?
  67. # [20:19] <tobie> ArtB: yes. We'd combine this with a bunch of automated tests ran on the submission (which the person merging the submission would have access to).
  68. # [20:19] <jgraham> ArtB: Well, I think the CR exit criteria are defined on a per-WG basis.
  69. # [20:20] <jgraham> So you can more or less choose the process you want there
  70. # [20:20] <jgraham> (all choices are bad though)
  71. # [20:24] <ArtB> well I am not aware of any Silver Bullets here
  72. # [20:26] <tobie> ArtB: there's a proposal to extend the notion of test facilitators for all specs in the web-platform-test repo.
  73. # [20:26] <tobie> ArtB: what if you made the facilitator responsible for deciding whether coverage is adequate or not?
  74. # [20:27] <tobie> ArtB: we'll have coverage data available too.
  75. # [20:27] <ArtB> well, we would need to quibble on the defn of "responsible"
  76. # [20:27] <ArtB> I think a TF should be qualified enuf to make a proposal re coverage
  77. # [20:27] <ArtB> but it's really a WG decision
  78. # [20:27] <tobie> ArtB: oh uh. You do that. :) I'm not part of WebApps. :)
  79. # [20:27] * Joins: AutomatedTester (~AutomatedTester@public.cloak)
  80. # [20:28] * Ms2ger doesn't care about CR exit criteria
  81. # [20:28] <ArtB> I hear all that time Ms2ger from Members with no IP ;-)
  82. # [20:28] <tobie> Ms2ger: that was a given.
  83. # [20:29] <jgraham> Well we have had the problem in WebApps where company A submits a testsuite for a feature and then company A's representative is asked "is your testsuite sufficient to go to CR?"
  84. # [20:29] <jgraham> Which, in the case that company A only cares about IP, is always answered "yes"
  85. # [20:29] <Ms2ger> And then two vendors submit their test suites, and suddenly duplicated tests
  86. # [20:30] <ArtB> so a fundamental problem here is certainly that different Members want different things from test suites
  87. # [20:30] * jgraham doesn't really care about duplicated tests
  88. # [20:30] <Ms2ger> ArtB, for all I care, you could publish recs from caniuse data
  89. # [20:30] * tobie doesn't care about what people don't care about.
  90. # [20:30] * tobie cares about what people care about.
  91. # [20:31] <Ms2ger> ArtB, that would allow people to work on interoperability testing without process pressure, and would get IP protection much sooner
  92. # [20:32] <tobie> heh
  93. # [20:32] * Ms2ger sees no downsides
  94. # [20:33] <jgraham> It would discourage some parties from submitting tests at all
  95. # [20:33] <Ms2ger> jgraham, so, git fetch w3c && git merge --ff-only, right?
  96. # [20:34] <jgraham> Yes, if you don't have any local changes
  97. # [20:34] <Ms2ger> No, doesn't work
  98. # [20:34] <jgraham> Where does it fail?
  99. # [20:34] <Ms2ger> $ git merge --ff-only
  100. # [20:34] <Ms2ger> usage: git merge [options] [<commit>...]
  101. # [20:34] <jgraham> Which git version?
  102. # [20:34] <jgraham> git --version
  103. # [20:34] <Ms2ger> 1.7.5.4
  104. # [20:35] <Ms2ger> --ff-only is listed in the help message
  105. # [20:35] <Ms2ger> And on the point of discouraging these parties; I'm not sure if that would be positive or negative overall
  106. # [20:36] <jgraham> Well they do sometimes contribute good tests
  107. # [20:36] <Ms2ger> News to me ;)
  108. # [20:36] <jgraham> So we have this problem
  109. # [20:36] <jgraham> Well we have lots of problems
  110. # [20:36] <Ms2ger> Amen
  111. # [20:37] <jgraham> I wonder if I am supposed to say this. Oh well.
  112. # [20:38] <jgraham> Microsoft are obviously heavily influenced by IPR concerns and seem to submit tests for that reason.
  113. # [20:38] <jgraham> Opera have submitted lots of tests in the past, but aren't developing their own engine anymore
  114. # [20:39] <jgraham> Mozilla submit fewer tests than one might hope for given their Misson (but can probably be encouraged to submit more tests)
  115. # [20:39] <jgraham> Google and Apple basically never submit tests
  116. # [20:39] <jgraham> So
  117. # [20:40] * ArtB noticed the "fire sale" on Opera tests ;-)
  118. # [20:40] * Joins: mdas (~mdas@public.cloak)
  119. # [20:40] <Ms2ger> Hi mdas :)
  120. # [20:41] <jgraham> Some serious work is needed to change this situation
  121. # [20:42] <jgraham> Ms2ger: Oh you probably want to say git merge --ff-only w3c/master
  122. # [20:42] <Ms2ger> Worked, thanks
  123. # [20:42] <Ms2ger> And yeah, I'm aware we're not doing as well as I'd like
  124. # [20:43] <jgraham> Otherwise there is a danger that we will end up with more specs where we have the minimum of tests needed to satisfy the Process
  125. # [20:43] <jgraham> but not enough to actually cover the corner cases
  126. # [20:43] <jgraham> Making the Process heavier isn't going to go down well (after all, everyone likes the IPR stuff)
  127. # [20:43] <Ms2ger> Aryeh's tried to get us to default to testharness.js, but that failed on "I don't like its API" grounds, unfortunately
  128. # [20:44] <ArtB> there is indeed a mismatch between caring about a set of tests that qualify all of the QA/customer requirements and a set that allows a spec to get IP commitments
  129. # [20:44] <ArtB> so dangerous in what way?
  130. # [20:44] <ArtB> is anyone blocking the deployment of their browser b/c the W3C test suite is not complete?
  131. # [20:45] <Ms2ger> Dangerous in the sense that the conformance test suite sucks
  132. # [20:45] <Ms2ger> And that interoperability sucks as a result
  133. # [20:45] <jgraham> Dangerous in the "interop. suffers and eventually the platform fails" kind of way (where ending up with a single rendering engine is a particular failure mode)
  134. # [20:46] <ArtB> well it must not suck enuf for the vendors James enumerated above to care
  135. # [20:47] <Ms2ger> Well
  136. # [20:47] <Ms2ger> Vendors benefit from having other vendors write tests
  137. # [20:47] <Ms2ger> Not so much from submitting tests themselves
  138. # [20:47] <mdas> Ms2ger: oh hello
  139. # [20:47] <jgraham> In the short term, submitting tests is a pain
  140. # [20:48] <jgraham> You have to move to someone else's setup for writing tests
  141. # [20:48] <jgraham> And inherit some of their problems (exposed as API decisions)
  142. # [20:49] <Ms2ger> (trivial PR up at https://github.com/w3c/web-platform-tests/pull/39)
  143. # [20:49] <jgraham> So for *today* it can seem like a bad idea
  144. # [20:50] <jgraham> And if you are relatively resource happy and don't really care about the long term success of the platform (or have good marketshare and only care about your own continued success) it can seem like giving a free leg up to your competitors
  145. # [20:50] <ArtB> so, the prevailing attitude here (when a browser group was in my building) was that W3C tests were a joke and they had to do ALL of the testing themselves. That attitude seems to be shared by most of the browser vendors in WebApps.
  146. # [20:50] <ArtB> I say most because Opera might see it differntly?
  147. # [20:51] <jgraham> Well the thing is
  148. # [20:51] <ArtB> (or perhaps "used to"
  149. # [20:51] <jgraham> The W3C doesn't have tests
  150. # [20:51] <Ms2ger> They are a joke :)
  151. # [20:51] <jgraham> People have tests
  152. # [20:51] <jgraham> That they contribute to the W3C pool
  153. # [20:51] <jgraham> (this would be a bit different if the W3C employed people to write tests, but they don't)
  154. # [20:51] <ArtB> but I do see some members willing to contribute to a "Test Suite for the Attorneys" ;-)
  155. # [20:51] <jgraham> The people with the most tests are the vendors
  156. # [20:52] <jgraham> But they largely aren't contributing them to the pool
  157. # [20:52] <jgraham> And where they are (excluding Opera), it is somewhat obvious that they have the lawyers in mind
  158. # [20:53] <tobie> Ms2ger: what parts of the IP was disliked would be useful info.
  159. # [20:53] <tobie> s/IP/API/
  160. # [20:53] <Ms2ger> tobie, it all happened in a public newsgroup
  161. # [20:53] * tobie is tired.
  162. # [20:53] <Ms2ger> tobie, mostly "more typing", IIRC
  163. # [20:53] <ArtB> so you talk about the "test suite for attorney" as if that's a Bad Thing
  164. # [20:53] <Ms2ger> As in, mochitests have ok/is
  165. # [20:54] <Ms2ger> And no test()
  166. # [20:54] <ArtB> another way to look at it is that without IP commitments, the OWP is at risk ;)
  167. # [20:54] <Ms2ger> Office of Water Programs? Oxford Wheels Project?
  168. # [20:55] <Ms2ger> Organizacja Wyzwolenia Palestyny?
  169. # [20:55] <tobie> Ms2ger: we could get the code's AST and rewrite the tests.
  170. # [20:55] <Ms2ger> tobie, as in, automatic rewriting from mochitest to th.js?
  171. # [20:56] <tobie> yeah
  172. # [20:56] <Ms2ger> That would lead to pretty unidiomatic th.js, I'm afraid :/
  173. # [20:56] <tobie> that, or write a shim.
  174. # [20:56] <tobie> Ms2ger: haven't looked at the code, so difficult to say.
  175. # [20:56] <tobie> links?
  176. # [20:57] <Ms2ger> The main issue is that th.js expects you to wrap logical blocks in test()
  177. # [20:57] <Ms2ger> Whereas mochitest just lets you write code in the global scope
  178. # [20:57] <tobie> Sure.
  179. # [20:58] <Ms2ger> (A smaller issue is that is/ok don't map cleanly to th.js assertions)
  180. # [20:58] <tobie> sounds like that implies one mochitest page -> one th.js test function
  181. # [20:59] <Ms2ger> (is does == instead of ===, and ok tests truthiness instead of ===true)
  182. # [20:59] <tobie> big deal.
  183. # [20:59] <tobie> We probably need both anyway.
  184. # [20:59] <Ms2ger> ==? Not in my experience
  185. # [21:00] <tobie> ok, yeah. That one sucks.
  186. # [21:00] <tobie> I'm actually more concerned about mapping these tests to specs.
  187. # [21:01] <Ms2ger> That's another problem, yes
  188. # [21:01] <Ms2ger> Also, using non-standard or cutting-edge JS
  189. # [21:01] <tobie> meh
  190. # [21:01] <tobie> AST rewrites.
  191. # [21:01] <Ms2ger> Might work, yes
  192. # [21:01] <Ms2ger> Though I'd like to see it for 'yield'
  193. # [21:02] <tobie> yeah, not to sure how that desugars. :-/
  194. # [21:06] * Joins: darobin (rberjon@public.cloak)
  195. # [21:07] * Quits: darobin (rberjon@public.cloak) (Client closed connection)
  196. # [21:09] <jgraham> I think trying to auto-rewrite mochitests is a terrible idea
  197. # [21:09] <tobie> Ms2ger: could you perhaps point to where in the Moz codebase I could find examples of those tests?
  198. # [21:09] <Ms2ger> tobie, which kind?
  199. # [21:09] <tobie> oh, man. how many kinds are there!?
  200. # [21:10] <Ms2ger> Mochitests, you mean?
  201. # [21:10] <tobie> jgraham: you're probably right.
  202. # [21:10] <tobie> Ms2ger: I suppose so.
  203. # [21:10] <tobie> jgraham: but it's a fun exercise.
  204. # [21:12] <Ms2ger> http://mxr.mozilla.org/mozilla-central/find?string=test_.*\.html%24&tree=mozilla-central&hint= :)
  205. # [21:13] <Ms2ger> You probably want the ones in /dom or /content, but not in /dom/imptests
  206. # [21:13] <Ms2ger> (The latter are, in fact, th.js tests)
  207. # [21:14] <Ms2ger> tobie, ^
  208. # [21:14] <tobie> ty
  209. # [21:15] * Quits: ArtB (~abarsto@public.cloak) ("Leaving.")
  210. # [21:18] * Joins: shepazu (schepers@public.cloak)
  211. # [21:24] <tobie> Ms2ger: It doesn't look that bad, actually. Even after an automatic transformation, there's be a significant amount of work triaging the tests.
  212. # [21:24] <tobie> Wonder if this something we should look into.
  213. # [21:25] <Ms2ger> Well, yes, I'm sure a lot of our tests just aren't very good :)
  214. # [21:26] <Ms2ger> Or not testing anything backed by a spec
  215. # [21:26] <Ms2ger> And then there's licensing \o/
  216. # [21:27] <tobie> Just because it's open-source doesn't mean yada yada.
  217. # [21:28] <tobie> oh well.
  218. # [21:28] <Ms2ger> See https://bugzilla.mozilla.org/show_bug.cgi?id=788511 for example
  219. # [21:29] <tobie> well, that was fun.
  220. # [21:30] <tobie> while it lasted.
  221. # [21:30] <tobie> Back to work now.
  222. # [21:30] <tobie> Evenin' folks.
  223. # [21:31] <Ms2ger> Night
  224. # [21:31] * Quits: glenn (~gadams@public.cloak) (Client closed connection)
  225. # [21:37] * Quits: tobie (tobie@public.cloak)
  226. # [21:59] * Joins: glenn (~gadams@public.cloak)
  227. # [22:10] * Quits: glenn (~gadams@public.cloak) (Client closed connection)
  228. # [22:10] * Joins: glenn (~gadams@public.cloak)
  229. # [22:12] * Joins: glenn_ (~gadams@public.cloak)
  230. # [22:12] * Quits: glenn (~gadams@public.cloak) (Client closed connection)
  231. # [22:27] * Quits: glenn_ (~gadams@public.cloak) (Client closed connection)
  232. # [22:28] * Joins: glenn (~gadams@public.cloak)
  233. # [22:46] * Quits: shepazu (schepers@public.cloak) ("is sleepy")
  234. # [22:54] * Quits: zcorpan (~zcorpan@public.cloak) (Client closed connection)
  235. # [22:56] * Joins: zcorpan_ (~zcorpan@public.cloak)
  236. # [23:09] * Quits: glenn (~gadams@public.cloak) (Ping timeout: 60 seconds)
  237. # [23:10] * Quits: plh (plehegar@public.cloak) ("Leaving")
  238. # [23:12] * Joins: glenn (~gadams@public.cloak)
  239. # [23:31] * Quits: Ms2ger (~Ms2ger@public.cloak) ("nn")
  240. # [23:48] * Joins: shepazu (schepers@public.cloak)
  241. # [23:51] * Quits: AutomatedTester (~AutomatedTester@public.cloak) (Client closed connection)
  242. # Session Close: Fri Mar 22 00:00:00 2013

The end :)