Sikuli: Using GUI Screenshots for Search and Automation
Proceedings of the 22nd annual ACM symposium on User interface software and technology, 2009 · paper · Computer science
Presents Sikuli, which lets users search help systems with a screenshot of a GUI element and automate interactions through screenshot-based scripts. A web-based study found screenshot search easy to learn and faster to specify than keywords.
5 findings
Substance Coverage reported as "an estimate of 70.5% (i.e., 361/500 dialogs had at least one relevant document)", but 361/500 = 72.2%.
- p. 5 Performance Evaluation, right column
obtained an estimate of 70.5% (i.e., 361/500 dialogs
- p. 5 Performance Evaluation, coverage
70.5% (i.e., 361/500 dialogs
- p. 5 Performance Evaluation, coverage
estimate of 70.5% (i.e., 361/500 dialogs
Only the authors can say what is right here.
Judge's note: Page 5 crop confirms the printed values. The same sentence ties the percentage directly to 361/500, and I found no other denominator. 70.5% of 500 is 352.5, which is not a whole number, so either the percentage or the count is wrong. Only the authors can say which.
Second check: another model agreed
Report a problem with this finding
Reference or label Text cites "salient elliptical patches (Figure 2.3) detected by the MSER detector", but in Figure 2 the ellipses are in panel "2. Visual Features"; panel 3 is "OCR of Embedded Text".
- p. 3 Screenshot Search Engine, right column
lient elliptical patches (Figure 2.3) detected by the MSER
- p. 3 Figure 2 panel headings
2. Visual Features
- p. 3 Screenshot Search Engine
elliptical patches (Figure 2.3) detected by the MSER
Judge's note: Full page 3 image: Figure 2 panel headings are '1. Surrounding Text', '2. Visual Features' (the panel with the elliptical patches) and '3. OCR of Embedded Text'. The pointer should be Figure 2.2.
Second check: another model agreed
Report a problem with this finding
Reference or label Reference 12 lists the third author as "U. Martin" ("Matas, J., O. Chum, U. Martin, and T. Pajdla"); the author is M. Urban.
- p. 11 Reference 12
Matas, J., O. Chum, U. Martin, and T. Pajdla
- p. 11 References, entry 12
Matas, J., O. Chum, U. Martin, and T. Pajdla.
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: the BMVC 2002 MSER paper's third author is Martin Urban (M. Urban). The page image of p.11 shows 'U. Martin' in reference 12, which looks like the given name and surname swapped.
Second check: another model agreed
Report a problem with this finding
Reference or label Reference 1 is dated "Commun. ACM 43(3), 107–114, 2003.", but volume 43 of CACM is dated 2000.
- p. 11 Reference 1
Commun. ACM 43(3), 107–114, 2003.
- p. 11 References 1
Commun. ACM 43(3), 107±114, 2003.
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: Communications of the ACM volume 43, issue 3 is March 2000. The page image of p.11 shows '43(3), 107–114, 2003.' Nothing else in the paper dates the article.
Second check: another model agreed
Report a problem with this finding
Wording "template matching based on normalized cross-validation [4]": template matching uses normalized cross-correlation.
- p. 6 Finding GUI Patterns on the Screen
plate matching based on normalized cross-validation [4]
- p. 6 Finding GUI Patterns on the Screen
normalized cross-validation [4]
- p. 6 Finding GUI Patterns on the Screen
template matching based on normalized cross-validation [4]
Judge's note: Rests on outside knowledge: the standard template-matching similarity measure is normalized cross-correlation, while cross-validation is a model-evaluation technique. The page 6 crop confirms the printed text 'normalized cross-validation'.
Second check: another model agreed
Report a problem with this finding
13 minor (typos and the like; most checked by the judge only)
Reference or label Reference 17 ("Prabaker, M., L. Bergman, and V. Castelli. An evaluation of using programming by demonstration...") is never cited in the body text.
- p. 11 Reference 17
Prabaker, M., L. Bergman, and V. Castelli
- p. 11 References, entry 17
Prabaker, M., L. Bergman, and V. Castelli.
Only the authors can say what is right here.
Judge's note: I searched the text layer for every bracketed citation and found [1]-[16] and [18]-[20], but no [17]. I also checked the related-work pages (p.10 and p.11) in the page images, and no [17] appears there. Only the authors can say where it was meant to be cited.
Second check: another model agreed
Report a problem with this finding
Wording "The most dramatic difference was familiarity (Q6) t(11) = 4.33, p < .001": the t value and the P value do not correspond under the declared test.
- p. 5 Results, last paragraph of left column
t(11) = 4.33, p < .001
- p. 5 Results, questionnaire paragraph
familiarity (Q6) t(11) = 4.33, p < .001
Only the authors can say what is right here.
Judge's note: Page 5 crop confirms the values. With 11 df, the two-sided critical t at .001 is about 4.437, so t = 4.33 gives a two-sided p of about .0012. The paper does not say the test was one-sided, and the other two t tests in the paragraph (3.87, p=.003; .76, p=.46) match two-sided values. Either reading is well below .05, so this is a query to the authors, not a change in significance.
Second check: another model agreed
Report a problem with this finding
Wording Scripts chain Pattern modifiers onto find() results ("for region in find(x).anySize().regions:", "find(...).similar(0.7)"), but the API section defines exact/similar/anyColor/anySize as methods of Pattern objects, and find() as returning regions.
- p. 9 Script 2, line 2
for region in find(x).anySize().regions:
- p. 7 Pattern subsection
An image-based pattern object has four methods
Only the authors can say what is right here.
Judge's note: Page 7 image: 'An image-based pattern object has four methods...' and 'Each method produces a new pattern', with find() returning regions or false. Page 9 image: Script 2 line 2 and Script 3 line 2 apply anySize()/similar() to find(...). The paper does not explain this usage, so it is a query for the authors.
Second check: another model agreed
Report a problem with this finding
Typo Ungrammatical phrase "that can be reliably detected in despite of variations in scale" (should be 'despite' or 'in spite of').
- p. 3 Screenshot Search Engine, right column
in despite of variations
- p. 3 Screenshot Search Engine, visual features paragraph
reliably detected in despite of variations in
Judge's note: Page 3 image, right column, confirms 'in despite of'.
Second check: none (judge only)
Report a problem with this finding
Typo Figure 6 caption: "small patterns of varing sizes (a)" should read 'varying'.
- p. 6 Figure 6 caption
varing sizes (a) and colors (b)
- p. 6 Figure 6 caption
varing sizes (a)
- p. 6 Figure 6 caption
small patterns of varing sizes (a)
Judge's note: Page 6 crop confirms 'varing'. The same caption spells 'varying' correctly later.
Second check: none (judge only)
Report a problem with this finding
Typo "The action commands specify what keyword and/or mouse events to be issued": 'keyword' should be 'keyboard'.
- p. 7 Action subsection
what keyword and/or mouse
- p. 7 Action subsection, first sentence
specify what keyword and/or mouse events
- p. 7 Action section
specify what keyword and/or mouse events to be issued
- p. 7
what keyword and/or mouse events to be issued
Judge's note: Page 7 image confirms the text. Everywhere else the paper speaks of mouse and keyboard events, and the commands listed (click, type) issue mouse and keystroke events.
Second check: none (judge only)
Report a problem with this finding
Typo Visual Dictionary example uses "d[Pattern(...).anysize()]", but the method is defined as "anySize()".
- p. 7 Visual Dictionary paragraph
.anysize()]
- p. 7 Pattern method list
anySize():
- p. 7 Visual Dictionary
d[Pattern(
- p. 7 Pattern methods list
anySize(): Allow matches of a different size
Judge's note: Page 7 image: the Pattern method list has 'anySize():' and the chaining example has '.anySize()', while the visual dictionary example shows lowercase '.anysize()'. Python is case-sensitive.
Second check: none (judge only)
Report a problem with this finding
Typo Script 2 listing line label "5. for x in patterns:" uses a period, while all other line labels use a colon (e.g. '4:' and '6:').
- p. 9 Script 2 listing, line 5
5. for x in patterns:
Judge's note: Page 9 image confirms '5.' where the other lines use 'N:'.
Second check: none (judge only)
Report a problem with this finding
Typo "the background that may change as the maker moves": 'maker' should be 'marker'.
- p. 9 Script 3 description
as the maker moves
- p. 9
may change as the maker moves
Judge's note: Page 9 image, Script 3 description, confirms 'maker'. The preceding sentences refer to the bus marker.
Second check: none (judge only)
Report a problem with this finding
Typo "deletes all visible Office files (Words, Excel, PowerPoint)": the application name is 'Word'.
- p. 9 Script 2 description
Office files (Words, Excel,
Judge's note: Page 9 image confirms 'Words' in a list of application names. Elsewhere the paper writes 'Word', e.g. 'a Word document icon' on p7.
Second check: none (judge only)
Report a problem with this finding
Typo "interact with any Python library to accomplish tasks neither can do it alone": the 'it' is redundant.
- p. 9 Example 5
tasks neither can do it alone
Judge's note: Page 9 image, Example 5 description, confirms the wording. This is a minor grammar slip.
Second check: none (judge only)
Report a problem with this finding
Typo "highlighting portions of the text as well application UI elements" is missing 'as' ('as well as').
- p. 10 Related Work, Help systems
as well application
Judge's note: Page 10 crop, Related Work right column, confirms the missing word.
Second check: none (judge only)
Report a problem with this finding
Typo "The later provided a basic set of scripting commands" should read "The latter", meaning VisScript as opposed to 'The former' (VisMap).
- p. 11 Related Work, Image-based interaction
The later
- p. 11 Related Work, VisMap/VisScript
The later provided a basic set of scripting commands
Judge's note: Page image p.11, left column: 'The former inferred ... The later provided ...'. The sentence contrasts two items, so 'latter' is meant.
Second check: none (judge only)