[Koha-devel] Deadlock issues when running the tests in parallel, and maybe not only in the tests?

[Victor Grousset/tuxayo]

Hi :)

IIUC this is where the CI calls "prove" to run the tests
https://gitlab.com/koha-community/koha-testing-docker/-/blob/master/files/run.sh#L213

And this is the place where KOHA_PROVE_CPUS is set.
https://gitlab.com/koha-community/koha-testing-docker/-/blob/master/files/run.sh#L155
It's not set in ".env" file so it should use "nproc" and this calls
"prove" with more than one thread.

Is that really the case? I tried locally use to "prove -j 4" and I'm
getting deadlocks with the database after a few minutes.

And do these deadlock in tests mean that Koha can hit them while running
in production?

I expect the tests to fail due to messing up with each other data but a
deadlock is something else.


Cheers,

[Joonas Kylmälä]

Hey Victor,

can you please check whether the $ENV{KOHA_TESTING} variable is set
correctly in your CI setup, and if it is not, does it fix the issue?
Please see the code in C4::Circulation::SendCirculationAlert(), it is
the only place where we do manual db locking as far as I know. It is
something we should fix by re-architecturing the message_queue database
table for digest messages. However, I'm interested in knowing if the
deadlocks now happen because of this and whether it is unit test
specific issue because at least based on the comments in
SendCirculationAlert() it appears to be so.

Regards,
Joonas

Post by Victor Grousset/tuxayo
Hi :)
IIUC this is where the CI calls "prove" to run the tests
https://gitlab.com/koha-community/koha-testing-docker/-/blob/master/files/run.sh#L213
And this is the place where KOHA_PROVE_CPUS is set.
https://gitlab.com/koha-community/koha-testing-docker/-/blob/master/files/run.sh#L155
It's not set in ".env" file so it should use "nproc" and this calls
"prove" with more than one thread.
Is that really the case? I tried locally use to "prove -j 4" and I'm
getting deadlocks with the database after a few minutes.
And do these deadlock in tests mean that Koha can hit them while running
in production?
I expect the tests to fail due to messing up with each other data but a
deadlock is something else.
Cheers,

--
Joonas Kylmälä
Tietojärjestelmäasiantuntija

Kansalliskirjasto
Kirjastoverkkopalvelut
PL 15 (Unioninkatu 36)
00014 Helsingin yliopisto
_______________________________________________
Koha-devel mailing list
Koha-***@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-com

[Joonas Kylmälä]

Hmm, based on the line
https://gitlab.com/koha-community/koha-testing-docker/-/blob/master/files/run.sh#L204
I would think the ENV variable is actually set so maybe it is not this.
Any more info on the tables it locks on?

Regards,
Joonas

Post by Joonas Kylmälä
Hey Victor,
can you please check whether the $ENV{KOHA_TESTING} variable is set
correctly in your CI setup, and if it is not, does it fix the issue?
Please see the code in C4::Circulation::SendCirculationAlert(), it is
the only place where we do manual db locking as far as I know. It is
something we should fix by re-architecturing the message_queue database
table for digest messages. However, I'm interested in knowing if the
deadlocks now happen because of this and whether it is unit test
specific issue because at least based on the comments in
SendCirculationAlert() it appears to be so.
Regards,
Joonas

Post by Victor Grousset/tuxayo
Hi :)
IIUC this is where the CI calls "prove" to run the tests
https://gitlab.com/koha-community/koha-testing-docker/-/blob/master/files/run.sh#L213
And this is the place where KOHA_PROVE_CPUS is set.
https://gitlab.com/koha-community/koha-testing-docker/-/blob/master/files/run.sh#L155
It's not set in ".env" file so it should use "nproc" and this calls
"prove" with more than one thread.
Is that really the case? I tried locally use to "prove -j 4" and I'm
getting deadlocks with the database after a few minutes.
And do these deadlock in tests mean that Koha can hit them while running
in production?
I expect the tests to fail due to messing up with each other data but a
deadlock is something else.
Cheers,

--
Joonas Kylmälä
Tietojärjestelmäasiantuntija

Kansalliskirjasto
Kirjastoverkkopalvelut
PL 15 (Unioninkatu 36)
00014 Helsingin yliopisto
_______________________________________________
Koha-devel mailing list
Koha-***@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.o

[Victor Grousset/tuxayo]

Hi :)

Post by Joonas Kylmälä
Hmm, based on the line
https://gitlab.com/koha-community/koha-testing-docker/-/blob/master/files/run.sh#L204
I would think the ENV variable is actually set so maybe it is not this.
Any more info on the tables it locks on?

I did yet processed your previous message but here quickly the error
messages. Important info, it has been ran on 19.11.x

t/db_dependent/Charset.t ................................................ ok
===( 4190;485 797/? 0/? 17/47 2/15
)============================DBD::mysql::db do failed: Deadlock found
when trying to get lock; try restarting transaction [for Statement
"DELETE FROM issues"] at t/db_dependent/Circulation/Branch.t line 49.
===( 4191;486 797/? 0/? 17/47 3/15
)============================DBD::mysql::db do failed: Deadlock found
when trying to get lock; try restarting transaction [for Statement
"DELETE FROM items"] at t/db_dependent/Circulation.t line 1174.
# No tests run!

# Failed test 'No tests run for subtest
"AllowRenewalIfOtherItemsAvailable tests"'
# at t/db_dependent/Circulation.t line 1269.
DBD::mysql::db do failed: Deadlock found when trying to get lock; try
restarting transaction [for Statement "DELETE FROM items"] at
t/db_dependent/Circulation.t line 1174.
# Looks like your test exited with 255 just after 18.
t/db_dependent/Circulation.t
............................................ Dubious, test returned 255
(wstat 65280, 0xff00)

[Mason James]

Post by Victor Grousset/tuxayo
Hi :)
IIUC this is where the CI calls "prove" to run the tests
https://gitlab.com/koha-community/koha-testing-docker/-/blob/master/files/run.sh#L213
And this is the place where KOHA_PROVE_CPUS is set.
https://gitlab.com/koha-community/koha-testing-docker/-/blob/master/files/run.sh#L155
It's not set in ".env" file so it should use "nproc" and this calls "prove" with more than one thread.
Is that really the case? I tried locally use to "prove -j 4" and I'm getting deadlocks with the database after a few minutes.

hi Victor

it is known behaviour with the test suite, i think its caused by the tests mocking database tables in parallel - which is something that would not happen during normal/production usage

you can see that all the database tests are forced to run sequentially...

   prove -j ${KOHA_PROVE_CPUS} --rules='seq=t/db_dependent/**.t'

https://gitlab.com/koha-community/koha-testing-docker/-/blob/master/files/run.sh#L213

_______________________________________________
Koha-devel mailing list
Koha-***@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.or

[Victor Grousset/tuxayo]

Thanks folks for the replies.
So I was running manually the test suite, and used -j from the CI but
didn't used --rules so that's why it was failing.