Options:
- # Session Start: Sat Feb 15 00:00:00 2014
- # Session Ident: #testing
- # [00:36] <bterlson_> gsnedders: I agree with two harnesses, but one for library testing (something really nice like Mocha would work) and one for language (something extremely simple, ie. the sputnik style test262 tests). Almost all jit tests fall into the language test bucket I think...
- # [00:40] <gsnedders> I'm unconvinced one can draw such a clear distinction between lang/lib. Consider arrays, which in some impls very much have different implementations of language-level operations.
- # [00:41] <bterlson_> Even if a test isn't designed to run in a loop doesn't mean you can't run it anyway and look for crashes. Or, run it a couple times in a baseline mode and then turn on jit and compare functional results. Other approaches exist too.
- # [00:41] <bterlson_> Just wrapping in a function doesn't force the tester to make their test function pure so doesn't seem to solve the problem... A number (that I don't recall) of test262 tests today are inside functions and don't run right if you execute them multiple times.
- # [00:41] <gsnedders> Looking for crashes isn't the interesting part, IMO. It's missing subtle semantics in JIT behaviour that's the big issue, IMO.
- # [00:42] <bterlson_> I agree
- # [00:42] <gsnedders> Sure, wrapping in a function doesn't force it: but it adds more of an expectation, esp. combined with a test harness that runs it multiple times by default
- # [00:42] <bterlson_> You are right it's a fuzzy line. Library testing is perhaps better described as API testing
- # [00:43] <bterlson_> other than the fact that there are some tests that simply can't be run multiple times, the harness could trivially auto-wrap and execute global scope tests multiple times as well...
- # [00:44] <gsnedders> I think having it in a function adds more of an expectation of it being run multiple times.
- # [00:45] <bterlson_> As much as adding documentation to that effect does I suppose
- # [00:45] <gsnedders> People don't read docs. :)
- # [00:45] <bterlson_> They will if they can't get their PR accepted until they do! :)
- # [00:45] <gsnedders> My experience is they don't until you explicit spell it out to them in the PR. :(
- # [00:46] * Quits: lmcliste_ (~lmclister@public.cloak) ("")
- # [00:46] <bterlson_> But, if what you said was true there wouldn't be non-pure test case functions in test262 today...
- # [00:46] <gsnedders> Right, there are levels of expectations. :)
- # [00:46] <bterlson_> I'll defer to your experience on that point
- # [00:47] <gsnedders> I've certainly missed subtle things when playing around with writeable/configureable getters/setters on Object.prototype, for example :)
- # [00:47] * Joins: lmcliste_ (~lmclister@public.cloak)
- # [00:47] * bterlson_ is now known as bterlson
- # [00:50] <bterlson> Sorry, which thread was your last statement continuing? :[
- # [00:50] <gsnedders> Gimme a minute, buying train tickets atm
- # [00:53] * Joins: ptressel (~chatzilla@public.cloak)
- # [00:55] * Quits: ptressel (~chatzilla@public.cloak) (Client closed connection)
- # [00:56] <gsnedders> bterlson: Purity of test functions… even when you know they're meant to be pure :)
- # [00:56] <bterlson> oh, yeah
- # [00:56] <bterlson> definitely
- # [00:58] <bterlson> I have run a number of existing test262 tests in loops, debugging why they don't work is often very difficult and subtle work
- # [01:00] * Quits: rhauck1 (~Adium@public.cloak) ("Leaving.")
- # [01:07] * Quits: plh (plehegar@public.cloak) ("Leaving")
- # [01:10] <gsnedders> I did it with all test262 tests a couple of years back :)
- # [01:10] <gsnedders> (Just testing if the second results was the same as the first)
- # [01:12] * Joins: rhauck (~Adium@public.cloak)
- # [01:17] <bterlson> Damn, we could have been doing the same thing at the same exact time... I hope now that test262 is on github more of this stuff will happen in public
- # [01:28] <gsnedders> Pretty certain what I did was little more than a five line hack :)
- # [02:11] * Quits: AutomatedTester (~AutomatedTester@public.cloak) (Client closed connection)
- # [02:19] * Joins: gitbot (~gitbot@public.cloak)
- # [02:19] -gitbot:#testing- [web-platform-tests] foolip opened pull request #635: Stop using /resources/media/sunflower.* (master...sunflower) https://github.com/w3c/web-platform-tests/pull/635
- # [02:19] * Parts: gitbot (~gitbot@public.cloak) (gitbot)
- # [02:55] * Quits: lmcliste_ (~lmclister@public.cloak) ("")
- # [03:00] * Quits: rhauck (~Adium@public.cloak) ("Leaving.")
- # [03:00] * Joins: ptressel (~chatzilla@public.cloak)
- # [03:23] * Quits: ptressel (~chatzilla@public.cloak) ("ChatZilla 0.9.90.1 [Firefox 27.0.1/20140212131424]")
- # [03:26] * Joins: ptressel (~chatzilla@public.cloak)
- # [06:54] * Joins: lmcliste_ (~lmclister@public.cloak)
- # [07:31] * Quits: lmcliste_ (~lmclister@public.cloak) ("")
- # [09:06] * Joins: mdyck (~mdyck@public.cloak)
- # [09:06] * Quits: mdyck (~mdyck@public.cloak) ("Leaving")
- # [11:34] * Joins: Ms2ger (~Ms2ger@public.cloak)
- # [11:54] * Joins: darobin (rberjon@public.cloak)
- # [12:32] * Joins: Lachy (~Lachy@public.cloak)
- # [13:10] * Quits: darobin (rberjon@public.cloak) ("Leaving...")
- # [14:20] * Quits: Lachy (~Lachy@public.cloak) ("My MacBook Pro has gone to sleep. ZZZzzz…")
- # [14:34] * Joins: Lachy (~Lachy@public.cloak)
- # [15:03] * Joins: abarsto (~abarsto@public.cloak)
- # [15:03] * abarsto is now known as ArtB
- # [15:04] * Joins: gitbot (~gitbot@public.cloak)
- # [15:04] -gitbot:#testing- [web-platform-tests] Ms2ger pushed 2 new commits to master: https://github.com/w3c/web-platform-tests/compare/084a1af9fab1...c8e4758706c3
- # [15:04] -gitbot:#testing- web-platform-tests/master 9ad5c6c Philip Jägenstedt: Stop using /resources/media/sunflower.*...
- # [15:04] -gitbot:#testing- web-platform-tests/master c8e4758 Ms2ger: Merge pull request #635 from foolip/sunflower...
- # [15:04] * Parts: gitbot (~gitbot@public.cloak) (gitbot)
- # [15:09] * ArtB is now known as ArtB_
- # [17:20] * Joins: gitbot (~gitbot@public.cloak)
- # [17:20] -gitbot:#testing- [web-platform-tests] foolip opened pull request #636: Merge and expand audio_004.htm and video_001.htm (master...autoplay) https://github.com/w3c/web-platform-tests/pull/636
- # [17:20] * Parts: gitbot (~gitbot@public.cloak) (gitbot)
- # [17:29] * Joins: gitbot (~gitbot@public.cloak)
- # [17:29] -gitbot:#testing- [web-platform-tests] foolip opened pull request #637: Remove the remaining overly helpful comments in media tests (master...rm-media-pre) https://github.com/w3c/web-platform-tests/pull/637
- # [17:29] * Parts: gitbot (~gitbot@public.cloak) (gitbot)
- # [17:54] * Quits: dauwhe (~dauwhe@public.cloak) (Client closed connection)
- # [18:03] * Joins: gitbot (~gitbot@public.cloak)
- # [18:03] -gitbot:#testing- [web-platform-tests] Ms2ger pushed 2 new commits to master: https://github.com/w3c/web-platform-tests/compare/c8e4758706c3...5e02b616200c
- # [18:03] -gitbot:#testing- web-platform-tests/master 0ceaadd Philip Jägenstedt: Remove the remaining overly helpful comments in media tests...
- # [18:03] -gitbot:#testing- web-platform-tests/master 5e02b61 Ms2ger: Merge pull request #637 from foolip/rm-media-pre...
- # [18:03] * Parts: gitbot (~gitbot@public.cloak) (gitbot)
- # [18:19] * Quits: ArtB_ (~abarsto@public.cloak) ("Leaving.")
- # [18:19] * Joins: abarsto (~abarsto@public.cloak)
- # [18:19] * abarsto is now known as ArtB
- # [19:33] * Quits: ptressel (~chatzilla@public.cloak) ("zzz")
- # [19:52] * Quits: Lachy (~Lachy@public.cloak) ("My MacBook Pro has gone to sleep. ZZZzzz…")
- # [21:07] * Joins: yankhates_cb (~yankhatescb@public.cloak)
- # [21:31] * Quits: Ms2ger (~Ms2ger@public.cloak) ("nn")
- # [21:44] * Joins: lmcliste_ (~lmclister@public.cloak)
- # [21:52] * Joins: AutomatedTester (~AutomatedTester@public.cloak)
- # [21:56] * Quits: shepazu (schepers@public.cloak) ("is sleepy")
- # [22:23] * Quits: lmcliste_ (~lmclister@public.cloak) ("")
- # [22:29] * Quits: yankhates_cb (~yankhatescb@public.cloak) (Ping timeout: 180 seconds)
- # [22:30] * Quits: anssik (~uid10742@public.cloak) (Ping timeout: 180 seconds)
- # [22:33] * Joins: anssik (~uid10742@public.cloak)
- # [22:39] * Joins: yankhates_cb (~yankhatescb@public.cloak)
- # [22:47] * Quits: yankhates_cb (~yankhatescb@public.cloak) (Ping timeout: 180 seconds)
- # [22:49] * Quits: AutomatedTester (~AutomatedTester@public.cloak) (Client closed connection)
- # [22:54] * Joins: yankhates_cb (~yankhatescb@public.cloak)
- # [23:02] * Joins: AutomatedTester (~AutomatedTester@public.cloak)
- # [23:22] * Quits: yankhates_cb (~yankhatescb@public.cloak) (Ping timeout: 180 seconds)
- # [23:31] * Joins: yankhates_cb (~yankhatescb@public.cloak)
- # [23:33] * Quits: ArtB (~abarsto@public.cloak) (Ping timeout: 180 seconds)
- # [23:40] * Quits: yankhates_cb (~yankhatescb@public.cloak) (Ping timeout: 180 seconds)
- # [23:48] * Joins: Lachy (~Lachy@public.cloak)
- # [23:49] * Joins: yankhates_cb (~yankhatescb@public.cloak)
- # Session Close: Sun Feb 16 00:00:01 2014
The end :)