/irc-logs / w3c / #testing / 2013-08-13 / end

Options:

# Session Start: Tue Aug 13 00:00:00 2013
# Session Ident: #testing
# [00:42] * Quits: tobie (tobie@public.cloak) (Ping timeout: 180 seconds)
# [01:03] * Joins: jhammel_ (~jhammel@public.cloak)
# [01:04] * Quits: jhammel (~jhammel@public.cloak) (Ping timeout: 180 seconds)
# [02:09] * Quits: ArtB (~abarsto@public.cloak) ("Leaving.")
# [02:12] * heycam is now known as heycam|away
# [02:13] * Quits: jhammel_ (~jhammel@public.cloak) (Ping timeout: 180 seconds)
# [02:20] * Quits: rhauck (~Adium@public.cloak) ("Leaving.")
# [02:20] * Joins: rhauck (~Adium@public.cloak)
# [02:22] * Joins: jhammel (~jhammel@public.cloak)
# [02:27] * Quits: rhauck (~Adium@public.cloak) (Ping timeout: 180 seconds)
# [03:20] * heycam|away is now known as heycam
# [04:14] * Quits: jhammel (~jhammel@public.cloak) ("leaving")
# [05:20] * Joins: gitbot (~gitbot@public.cloak)
# [05:20] -gitbot:#testing- [web-platform-tests] TTWF-Shanghai opened pull request #273: Create folders for TestTWF Shanghai. (master...submissions/TestTWF_Shanghai) https://github.com/w3c/web-platform-tests/pull/273
# [05:20] * Parts: gitbot (~gitbot@public.cloak) (gitbot)
# [08:43] * heycam is now known as heycam|away
# [09:24] * Joins: dom (dom@public.cloak)
# [09:31] * Joins: gitbot (~gitbot@public.cloak)
# [09:31] -gitbot:#testing- [web-platform-tests] Velmont closed pull request #273: Create folders for TestTWF Shanghai. (master...submissions/TestTWF_Shanghai) https://github.com/w3c/web-platform-tests/pull/273
# [09:31] * Parts: gitbot (~gitbot@public.cloak) (gitbot)
# [09:39] * Joins: zcorpan (~zcorpan@public.cloak)
# [10:37] * Quits: shepazu (schepers@public.cloak)
# [10:41] * Joins: shepazu (schepers@public.cloak)
# [10:42] * Quits: shepazu (schepers@public.cloak)
# [10:46] * Joins: darobin (rberjon@public.cloak)
# [10:53] * Joins: shepazu (schepers@public.cloak)
# [11:05] * heycam|away is now known as heycam
# [11:50] * Joins: tobie (tobie@public.cloak)
# [12:10] * Joins: abarsto (~abarsto@public.cloak)
# [12:10] * abarsto is now known as ArtB
# [12:48] * Joins: Ms2ger (~Ms2ger@public.cloak)
# [13:24] * Joins: darobin_ (rberjon@public.cloak)
# [13:30] * Quits: darobin (rberjon@public.cloak) (Ping timeout: 180 seconds)
# [13:30] * heycam is now known as heycam|away
# [14:31] * darobin_ is now known as darobin
# [16:38] * Joins: gitbot (~gitbot@public.cloak)
# [16:38] -gitbot:#testing- [web-platform-tests] jgraham closed pull request #133: Update Range tests for changes to the detach() method. (master...detached-range) https://github.com/w3c/web-platform-tests/pull/133
# [16:38] * Parts: gitbot (~gitbot@public.cloak) (gitbot)
# [16:56] * Joins: glenn (~gadams@public.cloak)
# [16:57] * Quits: zcorpan (~zcorpan@public.cloak) (Client closed connection)
# [16:57] * Joins: zcorpan (~zcorpan@public.cloak)
# [17:04] * Quits: zcorpan (~zcorpan@public.cloak) (Ping timeout: 180 seconds)
# [17:28] * Joins: zcorpan (~zcorpan@public.cloak)
# [17:40] * Joins: rhauck (~Adium@public.cloak)
# [17:53] <jgraham> tobie: https://github.com/jgraham/wptserve/commit/c05ca4534442d7c554d52410ed95b8876cd562f1
# [17:53] * Quits: darobin (rberjon@public.cloak) (Client closed connection)
# [17:53] <jgraham> Still not sure this is a good idea :)
# [17:53] <tobie> :D
# [17:53] * Joins: krisk (~krisk@public.cloak)
# [17:54] <tobie> I had something like that setup for PrototypeJS back in the days.
# [17:54] <tobie> Worked well.
# [17:54] <tobie> (though the reqs weren't as complicated)
# [17:54] * Joins: darobin_ (rberjon@public.cloak)
# [17:55] <jgraham> Hmm, given there is a typo in the commit, I wonder why that worked :)
# [17:55] <jgraham> Maybe I introduced the typo recently
# [17:56] <tobie> still think the pipe= is redundant
# [17:56] <jgraham> Tests sometimes require query strings
# [17:56] <jgraham> I don't want to reserve the whole of query-space for all these files
# [17:57] * Joins: kershaw (~kkershaw@public.cloak)
# [17:57] <krisk> Hello kershaw!
# [17:57] <kershaw> HI
# [17:57] <kershaw> HI
# [17:58] <Ms2ger> Hmm, is this a meeting day?
# [17:58] * Parts: kershaw (~kkershaw@public.cloak)
# [17:58] <krisk> Looks like a number of people are on the IRC channel, so we can chat about HTML stuff
# [17:58] <jgraham> I guess it must be
# [17:58] * Joins: mdyck_web (~mdyck_web@public.cloak)
# [17:58] <jgraham> Being in the UK makes this a nicer time :)
# [17:58] <krisk> Yes
# [17:59] <krisk> you welcome
# [17:59] <krisk> Looks like kershaw was online then was left, I suspect he has some questions?
# [18:00] <krisk> Agenda ->
# [18:00] <krisk> http://lists.w3.org/Archives/Public/public-html-testsuite/2013Aug/0011.html
# [18:01] <krisk> Their is a Test The Web Forward Event coming up on August 17th/18th
# [18:02] * Joins: kershaw (~kkershaw@public.cloak)
# [18:02] <krisk> It may create/should create some test submissions
# [18:02] <krisk> Hello again Kershaw!
# [18:03] <kershaw> Hi - very new to IRC - please bear with me
# [18:03] <krisk> No worries, we are nice :)
# [18:04] <krisk> Do you have any questions about creating/submitting tests?
# [18:04] <kershaw> Thanks - yes, lots.
# [18:04] <jgraham> :)
# [18:05] <krisk> Ms2Ger - I updated the wiki just for you to know when a meeting is going to occur
# [18:05] <krisk> http://www.w3.org/html/wg/wiki/Testing#Upcoming_meetings
# [18:05] <Ms2ger> Thanks :)
# [18:05] <krisk> kershaw feel free to ask any questions
# [18:05] <kershaw> I'm trying to organize a small team at CableLabs to do testing in some of the embedded content area -
# [18:06] <kershaw> some of you may have seen my earlier blast email a couple weeks ago.
# [18:06] <kershaw> I understand now that pending test submissions are PRs in Git.
# [18:06] <krisk> Yes, the list is also a fine place to ask questions
# [18:07] <kershaw> I probably need to get a prepared list of questions together for the next meeting so I apologize for unpreparedness at the moment
# [18:07] <tobie> hi folks
# [18:07] <Ms2ger> kershaw, if you have questions, ask them any time, no need to wait for a meeting
# [18:08] <krisk> You can also send mail to the list and say "hey can we meet on IRC #testing at X time?"
# [18:08] <kershaw> maybe the most useful thing for me would be to know who in this group is actively working to develop tests in the media area - specifically video, audio, track, & similar.
# [18:09] <kershaw> I understand Opera has a number of tests in the review queue. Is anyone else working actively in this territory?
# [18:09] <jgraham> kershaw: Usually with browser vendors tests are developed when the feature is implemented
# [18:10] * Quits: rhauck (~Adium@public.cloak) ("Leaving.")
# [18:10] <jgraham> So areas that are already well implemented are unlikely to get new tests from browser vendors, except possibly as a result of fixing bugs and/or conversion of existing tests to W3C format
# [18:11] <kershaw> so, tests are done once, submitted, approved over time, and generally, people don't circle back to look for coverage gaps?
# [18:11] * Joins: jhammel (~jhammel@public.cloak)
# [18:11] <kershaw> done once by a particular browser vendor, that is.
# [18:12] <jgraham> Well of course people try to get good coverage in the first place
# [18:12] <jgraham> and gaps are often exposed by implementation bugs
# [18:12] * Joins: rhauck (~Adium@public.cloak)
# [18:12] <jgraham> But many/most browser vendors don't seem to have a process that involves formally writing out each requirement and checking off that there is a test for it
# [18:13] <krisk> kershaw it also depends upon the feature at least in HTML
# [18:14] <krisk> For example everyone had a HTML parser, but surely everyone had to update their parser implementation and create tests to match the HTML5 spec
# [18:14] <jgraham> Right, that counts as "not well implemented"
# [18:14] <kershaw> fair enough. I thought one thing we might start with is to review any submitted but un-approved tests
# [18:14] <jgraham> kershaw: That would be * in Git.
# [18:14] <jgraham> 16:10 < krisk> Yes, the list is also a fine place to ask questions
# [18:14] <jgraham> Sigh
# [18:15] <jgraham> kershaw: that would be *tremendously* helpful
# [18:16] <jgraham> (mispaste before that)
# [18:17] <kershaw> I had planned to just work through the PRs, one at a time to look for those in our interest area. There's no shortcut around doing that is there?
# [18:17] <Ms2ger> Critic!
# [18:18] <jgraham> Indeed
# [18:18] <Ms2ger> https://critic.hoppipolla.co.uk/
# [18:18] <kershaw> sorry - no criticism intended
# [18:18] <Ms2ger> jgraham can sing its praises better than I can :)
# [18:18] <Ms2ger> "Critic" is the name of the tool
# [18:18] <jgraham> kershaw: Critic is a review tool
# [18:18] <kershaw> oh yeah - I remember now
# [18:18] <jgraham> You can set up filters to show you only reviews that you are interested in
# [18:19] <jgraham> So if you go to the main page (sign in using github credentials) and go to "Add filter"
# [18:19] <jgraham> Then you should select the web-platform-tests repo
# [18:19] <jgraham> and html/media/ or whatever the directory is as the path
# [18:20] <jgraham> (note: no leading slash, but a trailing slash)
# [18:20] * Quits: rhauck (~Adium@public.cloak) ("Leaving.")
# [18:20] <jgraham> then save the filter and go to "dashboard"
# [18:20] <jgraham> and you will see all the reviews that touch that folder
# [18:21] <kershaw> OK - thanks very much. We'll start on that
# [18:21] <krisk> kershaw this IRC channel is faster than the list, if you have a question in the future
# [18:23] <kershaw> so...you "guys" have the IRC channel open and monitor most all the time?
# [18:24] <Ms2ger> Some of us, at least
# [18:24] <jgraham> Well it depends who. But, for example Ms2ger and I are here much of the time that Europeans are likely to be awake
# [18:24] <kershaw> thanks - I had no clue. That's really helpful
# [18:27] <kershaw> This may be premature but are there any questions about what we're trying to do here that you have for me?
# [18:27] * Joins: Bin_Hu (~Bin_Hu@public.cloak)
# [18:27] <tobie> kershaw: think it would be worse discussing your requirements for finer-grained traceability than section ids.
# [18:27] <krisk> s/worse/worth/
# [18:27] * Ms2ger wonders what tobie is saying
# [18:27] <tobie> krijnh: ty
# [18:28] <tobie> arg
# [18:28] <tobie> Think I should just shut up.
# [18:28] <jgraham> Heh
# [18:28] <krisk> Kershaw: I am wondering more about requirements that you may have that may be outside of html?
# [18:28] <krisk> For example I suspect you'll need script support and css???
# [18:29] <kershaw> OK - so I think I can answer the 2nd question more easily.
# [18:29] * darobin_ is now known as darobin
# [18:29] * Quits: dom (dom@public.cloak) (Ping timeout: 180 seconds)
# [18:29] <jgraham> (fwiw, I think that the only fine-grained assertion metadata that I have seen that seemed to have a decent chance of working used id refs and regexps. That makes it relatively robust to spec changes, and makes it clearer which tests need to be updated when things do change)
# [18:30] * Parts: jhammel (~jhammel@public.cloak) (jhammel)
# [18:30] <jgraham> (solutions involving xpath are bad on many levels)
# [18:30] <kershaw> In general, cable operators in the US are helping me put a list of specs that they feel need support in the browsers for the HTML5 apps they're writing.
# [18:30] <kershaw> It's a big list and I can mail it out to the group.
# [18:31] <kershaw> That said, there are some areas, specifically around the elements I mentionned earlier that they're really "concerned" about. And si
# [18:32] <tobie> jgraham: kershaw, his team and I were discussing hashing conformance requirement sentences
# [18:32] <tobie> kershaw: making that document public would be useful
# [18:32] <kershaw> and since the general scope of "everything" they want is so massive, we decided to focus on those particular areas where existing cable services
# [18:32] <krisk> Kershaw: I think the big list would be intresting
# [18:33] <tobie> kershaw: public-test-infra@ is a good place for that.
# [18:33] <tobie> (despite the name)
# [18:33] <jgraham> tobie: Interesting idea
# [18:33] <kershaw> have to be mapped into the new HTML5 structures
# [18:33] <tobie> jgraham: unsure if the speed of change of a spec would make it ideal or impracticable.
# [18:34] <kershaw> those services are present in data structures of the MPEG-2 transport streams today
# [18:34] <jgraham> It would likely take a lot of work to keep up to date
# [18:35] <kershaw> WRT - requirement to test mapping, I could see I stepped in a minefield on my earlier post.
# [18:35] <tobie> jgraham: automation could help (file tasks, etc) but unsure if it would be enough
# [18:35] <tobie> kershaw: there are plenty of those. You'll get used to it.
# [18:36] <Ms2ger> If you don't step into minefields, you're doing something wrong :)
# [18:36] <mdyck_web> tobie: "hashing" in what sense?
# [18:36] <krisk> ms2ger: yes!
# [18:36] <Ms2ger> Probably because people don't care what you're doing :)
# [18:37] <tobie> mdyck_web: a hash function
# [18:37] <krisk> kershaw: You could also build your sample HTML5 app and they share what is not working across various browsers
# [18:37] <tobie> krisk: the effectiveness of such an approach is still tbd
# [18:38] <kershaw> In previous work (at CableLabs) we just enforced adding metadata to the test source to provide spec to test mapping and vice-versa
# [18:38] <Ms2ger> We're trying to encourage people to write tests for us despite not being paid to do so
# [18:38] <Ms2ger> So enforcing things people don't want to do is not easy
# [18:39] <tobie> ^ that
# [18:39] <kershaw> it's not even easy when you're paying people! :)
# [18:39] <tobie> I was about to say.
# [18:39] <krisk> kershaw: The reason I say this is alot of times stuff works quite well when all the browsers get the same markup
# [18:40] <krisk> No hacks, ua detection, etc...
# [18:40] <krisk> Also make sure your app doesn't use -webkit or other browser specific code
# [18:42] <kershaw> alright...I need to go off to a meeting so I'm going to disconnect. I will follow up with 2 things to the mailing list and let you guys beat on them
# [18:42] <Ms2ger> See you
# [18:43] <kershaw> One is the list-o-specs that cable operators think are critical to their HTML5 app development
# [18:43] <krisk> tobie: People don't have to share the full app, but indeed they could share the interop issue which could lead to a spec/implementation fixes
# [18:43] <jgraham> (ot: does anyone have a preference for how fully automatic handlers in the test server ought to work? Just a .py file with request and response as globals and no 'simple' API or a function e.g. def main(request, response): return content)
# [18:43] <jgraham> kershaw: Thanks, sounds like you are going to do valuable work
# [18:43] <krisk> simple is best with full control
# [18:44] <jgraham> krisk: If that was a response to me, they are both differently simple
# [18:44] <tobie> krisk: tried that with dedicated apps: e.g. coremob camera without any success whatsoever.
# [18:45] <tobie> krisk: took 18 month to gather vendor attention around appcache issues despite it plaguing the FB mobile app.
# [18:45] <jgraham> krisk: In one case you have to know what the hidden global variables are, and you have to interact with the response object (or a lower level) directly
# [18:45] <tobie> krisk: color me doubtful
# [18:45] <jgraham> In the other case you have to provide a known function name or something
# [18:45] <krisk> tobie: It did show that appcache had spec design issues...
# [18:46] <kershaw> Second is a proposal for test markup to provide that test <-> spec mapping. I'll be back later. Thanks all for your help and advice today.
# [18:46] <jgraham> but you can have a simpler API where you return stuff to use as status/headers/body (or just body, perhaps)
# [18:46] <krisk> tobie: it's very complex feature without alot tests...
# [18:46] <jgraham> I think I like the proposal with the function a bit better
# [18:47] <tobie> krisk: arguably. the coremob cam is showing blatant defaults in modern browsers and I'm not seeing more traction there.
# [18:47] <tobie> krisk: it's not a silver bullet
# [18:48] <Ms2ger> [Heh, coremob]
# [18:48] * Joins: dom (dom@public.cloak)
# [18:48] * Quits: kershaw (~kkershaw@public.cloak) (Client closed connection)
# [18:49] <krisk> jgraham: In my past I wrote a proxy server for fuzz testing HTTP/HTTPS
# [18:50] <krisk> For all the real intresting tests they needed to break all the rules, such that having full direct control of all headers and the response body was the most effective
# [18:50] <tobie> Ms2ger: [heh]
# [18:51] <krisk> For example the HTTP header had a content-length but the response body was 'chunked'
# [18:54] * Joins: rhauck (~Adium@public.cloak)
# [18:55] <jgraham> krisk: So the idea is that you should be able to break all the rules here
# [18:55] <jgraham> the response object has response.writer.write which is basically access to the raw socket
# [18:55] <jgraham> But for common cases you won't want to break *all* the rules
# [18:56] <krisk> yes
# [18:56] <krisk> maybe have the ability to go 'raw' but simple helper for most other cases
# [18:56] <jgraham> That is exactly the plan
# [18:56] <jgraham> There are various levels of API
# [18:57] <krisk> sounds fine to me...
# [18:57] <jgraham> You can interact with the writer directly to send headers + etc
# [18:57] <krisk> OK I have to take off for now...
# [18:57] <jgraham> Or you can use the default behaviour that will enforce normal HTTP semantics
# [18:57] <krisk> I'll take a peek at the IRC in a few hours...
# [18:58] <jgraham> And you provide a list of headers and a body
# [18:58] * Joins: kershaw (~kkershaw@public.cloak)
# [19:12] * Quits: ArtB (~abarsto@public.cloak) (Ping timeout: 180 seconds)
# [19:21] * Joins: abarsto (~abarsto@public.cloak)
# [19:22] * abarsto is now known as ArtB
# [19:33] * Quits: dom (dom@public.cloak) (Ping timeout: 180 seconds)
# [19:34] * Joins: bhill (~Brad@public.cloak)
# [19:34] * Quits: bhill (~Brad@public.cloak) (Client closed connection)
# [19:39] * Quits: darobin (rberjon@public.cloak) (Client closed connection)
# [19:44] * Quits: Bin_Hu (~Bin_Hu@public.cloak) ("Page closed")
# [20:06] <mdyck_web> tobie?
# [20:07] <tobie> yes?
# [20:07] <mdyck_web> re "hashing conformance requirement sentences"
# [20:09] <mdyck_web> so you'd take a spec, toss the non-normative chunks, and slice the rest into sentences, then push each of them through (say) md5sum to get a string of hex digits?
# [20:09] <tobie> yes
# [20:09] <tobie> then include a link to that in the test and in the spec itslef
# [20:10] <tobie> then have a bot going through the tests and specs and filing tickets when there are changes in the spec (this a hash change)
# [20:10] <tobie> s/this/thus/
# [20:11] <tobie> biab
# [20:11] <mdyck_web> you could do all that using the sentence text itself, rather than a hash of it
# [20:12] * Joins: jhammel (~jhammel@public.cloak)
# [20:20] <mdyck_web> moreover, using the sentence text: (1) the purpose of the test is clearer (you don't have to follow a link to see the requirement that it's testing)
# [20:26] <mdyck_web> I was going to say: "and (2) when the spec changes, it's easier to re-sync", but now I'm re-thinking that.
# [20:30] <mdyck_web> I think you'd want a bot that monitors the finest-granularity change-stream, and generates reports of the form:
# [20:35] * Joins: bhill (~Brad@public.cloak)
# [20:41] <mdyck_web> Commit/revision X of the spec [link to diff] changed or removed sentence Y, which is referenced by the following tests...
# [20:42] <mdyck_web> At that point, I think a human has to decide whether there
# [20:44] * Joins: zcorpan_ (~zcorpan@public.cloak)
# [20:44] * Quits: zcorpan (~zcorpan@public.cloak) (Client closed connection)
# [20:44] <mdyck_web> is a sentence in the revised spec that is normatively the same as sentence Y.
# [20:45] <Ms2ger> Sounds like work
# [20:46] <Ms2ger> But do I hear a volunteer? :)
# [20:47] <mdyck_web> If there is (i.e., sentence Y was just tweaked editorially), select that sentence, and press a button to automatically propagate the change to the spec-refs in the test suite.
# [20:48] * Ms2ger wants to see it
# [20:53] <mdyck_web> If there isn't a normative equivalent for sentence Y, then the resolution is tough, but you still want to automatically go to each test that references sentence Y and mark that the reference (though not necessarily the test) is invalid
# [20:57] <mdyck_web> (Formerly, this test tested [sentence Y], but that requirement no longer exists as such (as of commit X). Some human needs to determine if this is still a valid test, and if so, what requirement-sentence it is testing.)
# [21:07] <mdyck_web> Hm, for the first alternative, wouldn't have to be "normatively the same". E.g., if a spec sentence changes from "Every A must be P" to "Every A and every B must be P", that's not normatively the same (not simply an editorial change), but any tests that claim to be testing the former would also claim to be testing the latter
# [21:51] <mdyck_web> One fly in the ointment (of using sentence text or a hash thereof) is that a spec-sentence often doesn't fully (or uniquely) convey a requirement.
# [21:56] <mdyck_web> E.g., the HTML5 spec has 5 sentences whose text is "On getting, it must return the current value."
# [21:57] <mdyck_web> Each occurrence expresses a different requirement, but you can't distinguish them solely on the basis of the text of the sentence (or a hash of it).
# [21:58] <mdyck_web> Note that tossing in the section id doesn't help, because all 5 occurrences are in the same section (4.8.11.2.4 Line styles).
# [22:26] * Quits: rhauck (~Adium@public.cloak) ("Leaving.")
# [22:37] * Joins: rhauck (~Adium@public.cloak)
# [22:43] * Quits: Ms2ger (~Ms2ger@public.cloak) ("nn")
# [23:42] * heycam|away is now known as heycam
# [23:45] * Quits: zcorpan_ (~zcorpan@public.cloak) (Client closed connection)
# [23:45] * Joins: zcorpan (~zcorpan@public.cloak)
# [23:52] * Quits: zcorpan (~zcorpan@public.cloak) (Ping timeout: 180 seconds)
# Session Close: Wed Aug 14 00:00:00 2013

The end :)