Ticket #199 (closed task: fixed)

Opened 3 years ago

Last modified 2 weeks ago

Semeai improvements

Reported by: gunnar Owned by: gnugo
Priority: normal Milestone: 3.9.1
Component: source Version:
Severity: normal Keywords:
Cc: patch: yes

Description

This ticket is intended for collecting and evaluating patches to improve the semeai reading, in particular with regard to the STS-RV test collection (see #41). It will be closed when no further short term improvements are in sight.

Attachments

gunnar_7_12.7.diff Download (346 bytes) - added by gunnar 3 years ago.
Don't amalgamate dragons over three liberty strings. (Cf. #197)
gunnar_7_12.8.diff Download (494 bytes) - added by gunnar 3 years ago.
Don't allow sacrifice of goal worm when catching lunch in owl/semeai reading.
gunnar_7_12.7b.diff Download (346 bytes) - added by gunnar 3 years ago.
Don't amalgamate dragons over two liberty strings (probably not a good idea).
gunnar_7_12.9.diff Download (0.6 KB) - added by gunnar 3 years ago.
Don't play outer liberties in recognized eyespace.
gunnar_7_12.10.diff Download (0.5 KB) - added by gunnar 3 years ago.
Do some more distrusting of tactical moves in semeai reading.
gunnar_7_12.7c.diff Download (3.0 KB) - added by gunnar 3 years ago.
Don't amalgamate dragons over three liberty strings or a specific class of two liberty strings.
gunnar_7_12.8b.diff Download (1.1 KB) - added by gunnar 3 years ago.
Revised patch
gunnar_7_13.8.diff Download (2.7 KB) - added by gunnar 2 years ago.
Semeai testcase revisions.
gunnar_7_13.9.diff Download (2.2 KB) - added by gunnar 2 years ago.
Avoid overvaluation of eyespaces which are split by a single stone in atari, plus tuning.
gunnar_9_1.8.diff Download (29.4 KB) - added by gunnar 5 months ago.
Recognize eyefilling liberties in semeai.
gunnar_9_1.9.diff Download (2.6 KB) - added by gunnar 5 months ago.
Constrain D711 and D712 to only capture bent four in the corner strings.
gunnar_9_1.10.diff Download (1.3 KB) - added by gunnar 5 months ago.
Don't try to cut/connect stones which are already in the same string.
gunnar_9_1.11.diff Download (0.7 KB) - added by gunnar 5 months ago.
Disable ineffective and sometimes harmful owl pattern D1139.
gunnar_9_1.12.diff Download (0.5 KB) - added by gunnar 5 months ago.
Only try whether pass also gives seki at the root.
gunnar_9_1.13.diff Download (489 bytes) - added by gunnar 5 months ago.
Don't make domains for the secondary color, since this destroys more useful information in semeai reading.
gunnar_9_1.8b.diff Download (29.3 KB) - added by gunnar 5 months ago.
Same as gunnar_9_1.8, rediffed after gunnar_9_1.12.
gunnar_9_1.15.diff Download (0.8 KB) - added by gunnar 5 months ago.
Never consider backfilling move for common liberty inside own eyespace as risk free.
gunnar_9_1.16.diff Download (0.8 KB) - added by gunnar 5 months ago.
Don't try to cut/connect stones which are already in the same string, again.
gunnar_9_1.17.diff Download (5.0 KB) - added by gunnar 5 months ago.
Do not play common liberties after having already passed.
gunnar_9_1.18.diff Download (3.2 KB) - added by gunnar 5 months ago.
Remove redundant functionality to fill in opponent eyes in semeai.
gunnar_9_1.19.diff Download (0.6 KB) - added by gunnar 5 months ago.
Owl tuning
gunnar_9_1.20.diff Download (463 bytes) - added by gunnar 5 months ago.
Play common liberties in semeai even if tactically unsafe in certain situations.
gunnar_9_1.21.diff Download (0.8 KB) - added by gunnar 5 months ago.
Capture sufficiently big nakade when critical semeai worm in atari, even if tactical reading disagrees about safety.
gunnar_9_1.22.diff Download (9.8 KB) - added by gunnar 5 months ago.
Breaking out new function test_lunch_essentiality() from owl_find_lunches(). Is not supposed to make any functional changes.
gunnar_9_1.23.diff Download (6.4 KB) - added by gunnar 5 months ago.
Allow tactically unstable owl lunches to be inessential.
gunnar_9_1.24.diff Download (497 bytes) - added by gunnar 5 months ago.
Small bonus to vital attack moves over vital defense moves in owl reading.
gunnar_9_1.25.diff Download (15.0 KB) - added by gunnar 4 months ago.
Testcase revision for STS-RV_1 and STS-RV_e.

Regression Results

Attachment Rev. PASS FAIL Nodes Status
gunnar_7_12.10.diff Download 2410 6 5 -0.15% -0.73% -0.049% details
2411 6 5 -0.17% -0.74% -0.051% details
2417 6 5 -0.15% -0.7% -0.034% details
2418 6 6 -0.16% -0.72% -0.036% details
2430 7 7 -0.19% -0.83% -0.082% details
2431 9 7 -0.13% -0.68% -0.025% details
2435 9 7 -0.13% -0.62% -0.026% details
2452 9 7 -0.12% -0.58% -0.027% details
2456 6 6 -0.16% -0.72% -0.071% details
2460 6 6 -0.15% -0.73% -0.071% details
2463 8 7 -0.16% -0.77% -0.069% details
2469 9 6 -0.13% -0.71% -0.054% details
gunnar_7_12.7.diff Download 2410 +0.089% +0.21% -0.041% details
gunnar_7_12.7b.diff Download 2410 6 5 +0.36% +0.71% +0.021% details
gunnar_7_12.7c.diff Download 2410 +0.089% +0.21% -0.041% details
gunnar_7_12.8.diff Download 2410 4 14 +0.1% -0.03% +0.02% details
2417 4 14 +0.11% -0.011% +0.02% details
2418 18 16 +0.11% -0.038% +0.02% details
2430 19 16 +0.096% -0.03% +0.01% details
gunnar_7_12.8b.diff Download 2410 5 5 +0.11% +1.5% -0.21% details
2411 5 5 +0.097% +1.5% -0.21% details
2417 5 5 +0.051% +1.1% -0.21% details
2418 19 7 +0.049% +1.1% -0.21% details
2430 19 8 +0.07% +1.3% -0.23% details
gunnar_7_12.9.diff Download 2410 1 2 +0.017% +0.07% +0.013% details
2411 1 2 +0.019% +0.071% +0.013% details
2417 1 2 +0.012% +0.045% +0.0045% details
2418 16 11 +0.012% +0.055% +0.0045% details
2430 18 9 +0.0048% +0.036% -0.018% details
2431 16 11 +0.012% -0.0073% -0.033% details
2435 17 10 +0.012% -0.033% -0.027% details
gunnar_7_13.8.diff Download never tested
gunnar_7_13.9.diff Download 2432 9 +0.11% +0.19% +0.022% details
gunnar_9_1.10.diff Download 2449 1 +0.0005% +0.0054% -0.0014% details
gunnar_9_1.11.diff Download 2449 -0.42% -0.18% +0.01% details
gunnar_9_1.12.diff Download 2452 2 1 -0.014% -0.18% +0.033% details
gunnar_9_1.13.diff Download 2452 1 2 -0.016% +0.016% +0.013% details
gunnar_9_1.15.diff Download 2456 7 1 -0.001% +0.085% -0.0072% details
gunnar_9_1.16.diff Download 2460 2 1 -0.035% -0.012% 0% details
gunnar_9_1.17.diff Download 2461 5 1 -0.015% -0.21% +0.0033% details
gunnar_9_1.18.diff Download 2461 +0.0072% +0.02% +0.0037% details
gunnar_9_1.19.diff Download 2463 1 +0.0033% -0.0035% +0.0033% details
gunnar_9_1.20.diff Download 2463 1 +0.0024% +0.015% +1.6e-05% details
gunnar_9_1.21.diff Download 2463 2 2 -0.00037% +0.0096% -0.0004% details
gunnar_9_1.22.diff Download 2466 0% 0% 0% details
gunnar_9_1.23.diff Download 2467 4 +0.0095% -0.19% -0.0025% details
gunnar_9_1.24.diff Download 2467 2 1 +0.072% +0.086% +0.037% details
gunnar_9_1.25.diff Download never tested
gunnar_9_1.8.diff Download 2449 26 11 -0.081% +0.097% -0.061% details
2452 27 14 -0.095% -0.22% -0.059% details
gunnar_9_1.8b.diff Download 2454 27 17 -0.072% -0.24% -0.052% details
gunnar_9_1.9.diff Download 2449 2 -0.023% -0.096% -0.02% details

Change History

Changed 3 years ago by gunnar

Don't amalgamate dragons over three liberty strings. (Cf. #197)

Changed 3 years ago by gunnar

Don't allow sacrifice of goal worm when catching lunch in owl/semeai reading.

follow-up: ↓ 3   Changed 3 years ago by gunnar

There is no breakage on the standard regressions for gunnar_7_12.7 and a small increase in reading nodes (+0.089% +0.21% -0.041%). On STS-RV the result is five passes and no fails. The passes are

STS-RV_1:23     PASS 1 0 C3 [1 0 (C3|C2)]
STS-RV_GSAT:26  PASS 1 1 G2 [1 1 (E3|G2|F1)]
STS-RV_Misc:16  PASS 1 1 A18 [1 1 (B19|C19|A18)]
STS-RV_Misc:19  PASS 1 1 Q5 [1 1 (T5|S5|R5|Q5|O1)]
STS-RV_RH:18    PASS 0 0 PASS [0 0 PASS]

All these results are relative r2410.

Changed 3 years ago by gunnar

Don't amalgamate dragons over two liberty strings (probably not a good idea).

Changed 3 years ago by gunnar

Don't play outer liberties in recognized eyespace.

Changed 3 years ago by gunnar

Do some more distrusting of tactical moves in semeai reading.

follow-up: ↓ 4   Changed 3 years ago by gunnar

There's a rather bad breakage for gunnar_7_12.8 on the normal regressions:

owl:123         PASS 1 C5 [1 C5]
trevorb:590     PASS M9 [H12|M9]
semeai:73       FAIL 1 1 PASS [1 0 PASS]
semeai:74       FAIL 0 0 PASS [1 0 (PASS|L13)]
semeai:75       FAIL 1 1 PASS [1 0 PASS]
semeai:76       FAIL 0 0 PASS [1 0 (PASS|N3|K2)]
semeai:77       FAIL 1 1 PASS [1 0 PASS]
semeai:78       FAIL 0 0 PASS [1 0 (PASS|A6|C6|C1)]
semeai:79       FAIL 1 1 PASS [1 0 PASS]
semeai:80       FAIL 0 0 PASS [1 0 (PASS|E9|E8|J9|H5)]
owl1:279        PASS 1 PASS [1 PASS]
century2002:180 FAIL O11 [T9]
auto_handtalk:3 FAIL 1 T15 [2 T15]
gunnar:101      PASS T8 [T8]
thrash:23       FAIL J6 [A5|D7]
13x13b:21       FAIL J13 [K11]
seki:1106       FAIL C9 [C1]
seki:2050       FAIL dead [alive]
4 PASS
14 FAIL
Total nodes: 1704626871 3334488 12416594 (+0.1% -0.03% +0.02%)

For STS-RV it's rather the opposite, 14 passes, 2 fails:

STS-RV_1:3      PASS 1 1 D8 [1 1 (D8|C3|B3|A3)]
STS-RV_1:5      PASS 1 1 M4 [1 1 (M4|N4|R3|R2|R1)]
STS-RV_1:20     PASS 1 1 T3 [1 1 (R3|S4|S2|T3)]
STS-RV_1:54     PASS 1 0 O2 [1 0 (O2|O1|R2|Q1|S1)]
STS-RV_1:59     PASS 0 0 PASS [0 0 PASS]
STS-RV_1:158    PASS 1 1 P6 [1 1 (N3|O5|P6|Q6|R6|S6|T6)]
STS-RV_1:160    FAIL 0 0 PASS [1 1 (N17|017|O16|O15|O14|R13|T12)]
STS-RV_1:205    PASS 1 1 C2 [1 1 C2]
STS-RV_Misc:4   PASS 1 1 A10 [1 1 A10]
STS-RV_Misc:9   PASS 1 1 J1 [1 1 J1]
STS-RV_Misc:14  PASS 1 0 B9 [1 0 B9]
STS-RV_Misc:35  PASS 1 1 J1 [1 1 J1]
STS-RV_Misc:42  PASS 1 0 A9 [1 0 A9]
STS-RV_Misc:44  PASS 1 0 B9 [1 0 B9]
STS-RV_RH:67    FAIL 0 0 PASS [1 1 B4]
STS-RV_e:2      PASS 1 1 B17 [1 1 B17]

Changed 3 years ago by gunnar

Don't amalgamate dragons over three liberty strings or a specific class of two liberty strings.

in reply to: ↑ 1   Changed 3 years ago by gunnar

Replying to gunnar:

There is no breakage on the standard regressions for gunnar_7_12.7 and a small increase in reading nodes (+0.089% +0.21% -0.041%). On STS-RV the result is five passes and no fails.

The patch gunnar_7_12.7b has a mixed breakage with 6 passes and 5 fails on the normal regressions plus increasing the owl nodes by 0.7%. On STS-RV it adds one more pass for STS-RV_1:58.

The more complicated patch gunnar_7_12.7c retains the pass for STS-RV_1:58 but without any breakage on the normal regressions and the same node cost as gunnar_7_12.7.

in reply to: ↑ 2   Changed 3 years ago by gunnar

Replying to gunnar:

There's a rather bad breakage for gunnar_7_12.8 on the normal regressions:

The typical problem can be examplified with semeai:73. In this position








the patch changes the three white stones from a lunch to an eyespace. As a consequence black is found to have two eyes and claims life, whereas white doesn't look alive, so the semeai is statically evaluated as a full win for black.

Changed 3 years ago by gunnar

Revised patch

  Changed 2 years ago by gunnar

  • milestone changed from 3.7.12 to 3.7.13

  Changed 2 years ago by gunnar

semeai:133      FAIL 1 1 H2 [1 2 H2]

This is actually backwards. The correct result is 1 1 H2!.

  Changed 2 years ago by gunnar

nngs2:150       FAIL J18 [M3|L3]

The semeai reading for the upper left corner is nowhere near correct with or without the patch gunnar_7_12.9.

  Changed 2 years ago by gunnar

gunnar_7_12.8b makes a huge difference when it comes to avoiding to play supposedly lunch-capturing moves inside own eyespace, e.g. in STS-RV_e:2. That combined with a positive overall regression balance suffices for accepting it.

Changed 2 years ago by gunnar

Semeai testcase revisions.

Changed 2 years ago by gunnar

Avoid overvaluation of eyespaces which are split by a single stone in atari, plus tuning.

  Changed 19 months ago by gunnar

  • milestone changed from 3.7.13 to 3.9.1

Changed 5 months ago by gunnar

Recognize eyefilling liberties in semeai.

  Changed 5 months ago by gunnar

The patch gunnar_9_1.8.diff expands on gunnar_7_12.9.diff by finding which liberties inside an eyespace can be filled in without increasing the eyevalue of the eyespace.

Changed 5 months ago by gunnar

Constrain D711 and D712 to only capture bent four in the corner strings.

Changed 5 months ago by gunnar

Don't try to cut/connect stones which are already in the same string.

Changed 5 months ago by gunnar

Disable ineffective and sometimes harmful owl pattern D1139.

  Changed 5 months ago by gunnar

gunnar_9_1.[9,10,11] have all been committed. gunnar_7_12.9 is obsoleted by gunnar_9_1.8.

Changed 5 months ago by gunnar

Only try whether pass also gives seki at the root.

Changed 5 months ago by gunnar

Don't make domains for the secondary color, since this destroys more useful information in semeai reading.

  Changed 5 months ago by gunnar

These are the results of gunnar_9_1.12:

STS-RV_1:63     PASS 1 0 PASS [1 0 PASS]
STS-RV_1:67     FAIL 1 0 B14 [1 1 (B14|C14|A17|B17|C17|B16)]
STS-RV_1:68     PASS 1 0 K19 [1 0 (G15|H15|K19|K18|G17|G19|F18|H18)]

In 63 the reading is reasonable without the patch but runs out of semeai nodes and fails. With the patch the reading is similar but finishes correctly well before running out of nodes. In 67 and 68 the reading both with and without the patch is way off, badly needing the changes in gunnar_9_1.8. Both the fail of 67 and pass of 68 are random results of running out of nodes without the patch but not with the patch.

  Changed 5 months ago by gunnar

These are the results of gunnar_9_1.13:

semeai:47       FAIL 1 1 T11 [1 0 (PASS|T11|T9|R9|T10|R10|R11)]
semeai:120      FAIL 1 0 A14 [0 0 PASS]
strategy4:166   PASS D11 [D11]

In semeai:47 two key moves are missed without the patch but only one with the patch. Since the mistakes no longer cancel the result is a fail.

Semeai:120 is more or less the same as semeai:47, more specifically, after playing up to this position,

   A B C D E F G H J K L M N O P Q R S T
19 . X . . X X . O X . . . . . . . . . . 19
18 O O X X X X X X O X X . O O . O O X . 18
17 . X O X O O O O O . . O . X O O X . . 17
16 O O O X X O X . . X X O X X X X X . . 16
15 . O O O O X X O O O O X O . . . . . . 15
14 X X O X X X . . . . O X . . . . O . . 14
13 . X X . . . . . . . . X O . . . . . . 13
12 . . . . . . . . . X . . X . . O . . . 12

tactical reading mistakenly thinks A18 can be defended at A17 (only C19 works as tactical defense), making white play A17 both with and without patch. After that black misses the key point without the patch, playing A15 and white get C19 in the following move. With the patch black correctly plays C19 and the end result is a fail.

strategy4:166 is mostly confusing but it is no longer "changing status of H18 from critical to alive" based on semeai results so there is probably a real improvement.

Changed 5 months ago by gunnar

Same as gunnar_9_1.8, rediffed after gunnar_9_1.12.

  Changed 5 months ago by gunnar

Below is a review of the breakage from gunnar_9_1.8b. To summarize there are 27 PASS, 17 FAIL and -0.24% owl nodes. There are many cases of major improvements of the reading and even most failures show a neutral or improved reading. There is no case of a really bad failure.

nngs:820        FAIL D14 [J13|L9]
Slightly worse, failing after running out of nodes.

semeai:134      PASS 1 1 H2 [1 1 H2]
This shows no difference on its own and the pass is apparently caused
by persistent cache interference from semeai:133.

STS-RV_1:32     FAIL 1 0 B4 [1 0 PASS]
B4 is also acceptable.

STS-RV_1:62     FAIL 1 1 T3 [1 0 T3]
Reading improved, despite the fail.

STS-RV_1:66     FAIL 0 0 PASS [1 0 (.*)]
Reading improved, despite the fail. Very similar to STS-RV_1:62.

STS-RV_1:75     PASS 1 1 C18 [1 1 (C18|B17|D17)]
Major improvement, reading the semeai correctly.

STS-RV_1:79     PASS 1 1 E19 [1 1 (E19|F18|F17)]
Major improvement, reading the semeai correctly.

STS-RV_1:81     PASS 1 1 E1 [1 1 E1]
Major improvement, reading the semeai correctly.

STS-RV_1:85     PASS 1 1 L16 [1 1 (N17|L16)]
Major improvement, reading the semeai correctly.

STS-RV_1:86     PASS 1 1 R16 [1 1 (Q17|R16)]
Major improvement, reading the semeai correctly.

STS-RV_1:87     PASS 1 1 D17 [1 1 (B17|D17)]
Major improvement, reading the semeai correctly.

STS-RV_1:89     PASS 1 1 S6 [1 1 (T5|S6)]
Major improvement, reading the semeai correctly.

STS-RV_1:93     FAIL 1 1 S6 [0 0 PASS]
Big improvement of reading but fail after running out of nodes.

STS-RV_1:96     PASS 0 0 PASS [0 0 PASS]
Major improvement, reading the semeai correctly.

STS-RV_1:98     PASS 0 0 PASS [0 0 PASS]
Major improvement, reading the semeai correctly.

STS-RV_1:119    PASS 1 1 C2 [1 1 C2]
Major improvement, reading the semeai correctly.

STS-RV_1:140    PASS 0 0 PASS [0 0 PASS]
Major improvement, reading the semeai correctly.

STS-RV_1:155    FAIL 0 0 PASS [1 1 C2]
Worse reading, probably change in move order causes interference by
tactical reading.

STS-RV_1:163    PASS 1 1 C18 [1 1 (C18|D17)]
Major improvement, reading the semeai correctly but somewhat inefficiently.

STS-RV_1:164    PASS 1 0 C18 [1 0 C18]
Major improvement, reading the semeai correctly but somewhat inefficiently.

STS-RV_1:166    PASS 1 1 D2 [1 1 D2]
Major improvement, reading the semeai correctly.

STS-RV_1:173    PASS 1 1 D17 [1 1 D17]
Major improvement, reading the semeai correctly but somewhat inefficiently.

STS-RV_1:184    PASS 1 1 C11 [1 1 C11]
Major improvement, reading the semeai correctly but somewhat inefficiently.

STS-RV_1:186    PASS 1 1 J2 [1 1 (J2|B2|B3|D3)]
Major improvement, reading the semeai correctly.

STS-RV_1:195    PASS 1 1 K2 [1 1 K2]
Major improvement, reading the semeai correctly.

STS-RV_1:196    PASS 1 1 D3 [1 1 D3]
Major improvement, reading the semeai correctly but somewhat inefficiently.

STS-RV_1:197    PASS 1 1 T1 [1 1 (T4|S3|T1)]
Major improvement, reading the semeai correctly.

STS-RV_Misc:7   FAIL 1 0 N7 [1 1 N7]
Much improved semeai reading, unlucky failure.

nngs2:150       FAIL J18 [M3|L3]
Arbitrary result. The semeai reading is way off both with and without
the patch.

ninestones:570  FAIL A2 [A9|B10|D11|F11|J15|A16]
Testcase looks questionable, is not black already safe? In any case
the reading does not look any worse than before.

ninestones:760  FAIL J11 [J10]
The position is a tangle of interrelated semeais. I can't quite find
out which specific semeai reading causes the difference but it's safe
to say that GNU Go just doesn't understand the position.

seki:112        PASS B1 [B1]
Possibly somewhat worse reading.

9x9:250         PASS A2 [A2|C1|C2]
Possibly minor improvement.

STS-RV_1:63     fail
Worse reading, somewhat unlucky.

thrash:23       pass
Random, running out of nodes both with and without patch.

STS-RV_Misc:8   fail
Improved semeai reading but still far from accurate.

STS-RV_1:77     pass
Major improvement but still a bit suspicious.

nngs:304        pass
Much improved.

STS-RV_1:67     pass
Major improvement, reading the semeai almost correctly.

STS-RV_1:5      fail
Better reading but not good enough.

STS-RV_Misc:42  fail
Terrible reading both with and without patch.

STS-RV_1:54     fail
Bad reading both with and without patch.

STS-RV_1:68     fail
Much improved reading but still not good enough.

STS-RV_Misc:44  fail
Mostly same reading as STS-RV_Misc:42.

Changed 5 months ago by gunnar

Never consider backfilling move for common liberty inside own eyespace as risk free.

Changed 5 months ago by gunnar

Don't try to cut/connect stones which are already in the same string, again.

Changed 5 months ago by gunnar

Do not play common liberties after having already passed.

Changed 5 months ago by gunnar

Remove redundant functionality to fill in opponent eyes in semeai.

Changed 5 months ago by gunnar

Owl tuning

Changed 5 months ago by gunnar

Play common liberties in semeai even if tactically unsafe in certain situations.

Changed 5 months ago by gunnar

Capture sufficiently big nakade when critical semeai worm in atari, even if tactical reading disagrees about safety.

  Changed 5 months ago by gunnar

The patch gunnar_9_1.21 has the following regression results:

STS-RV_Misc:9   fail
STS-RV_1:160    pass
STS-RV_Misc:35  fail
STS-RV_Misc:16  pass

In all cases the reading is improved by the patch.

Changed 5 months ago by gunnar

Breaking out new function test_lunch_essentiality() from owl_find_lunches(). Is not supposed to make any functional changes.

Changed 5 months ago by gunnar

Allow tactically unstable owl lunches to be inessential.

Changed 5 months ago by gunnar

Small bonus to vital attack moves over vital defense moves in owl reading.

Changed 4 months ago by gunnar

Testcase revision for STS-RV_1 and STS-RV_e.

  Changed 2 weeks ago by gunnar

  • status changed from new to closed
  • resolution set to fixed

Enough is enough. Let's close this ticket. All patches have either been added to the repository or become obsolete, with the exception of gunnar_7_12.10, which has been transferred to ticket #224.

Note: See TracTickets for help on using tickets.