You are not logged in.
We are having performance problems on our ReportServer server and are thinking it might be related to numerous scheduled jobs that appear to be stuck "In Execution".
The jobs started yesterday (multiple instances at different times) and all show as "In Execution". We tried to archive the jobs in the UI, and while any new instances were removed/not started, we still see those already started instances showing as "In Execution".
When we restart Tomcat, it takes a very long time (25 minutes), as it seems like the scheduled jobs are trying to run the reports even before ReportServer has finished starting up (or at least that's what the order of the logs in catalina.out suggest). We do not have any new / active scheduled jobs, so we're guessing it is the "In Execution" ones that are doing something (they never seem to finish though).
I was able to clean up these jobs by running a startup script as root that would remove the report itself, which caused all jobs to be removed as well. On the next restart, catalina.out had a number of entries like this, but was then faster:
WARNING: job execution failed
net.datenwerke.scheduler.service.scheduler.exceptions.JobExecutionException: Could not find report with id: null
at net.datenwerke.rs.scheduler.service.scheduler.jobs.report.ReportExecuteJob.doExecute(ReportExecuteJob.java:114)
at net.datenwerke.scheduler.service.scheduler.entities.AbstractJob.execute(AbstractJob.java:185)
at net.datenwerke.scheduler.service.scheduler.tasks.SchedulerTask.executeJob(SchedulerTask.java:527)
at net.datenwerke.scheduler.service.scheduler.tasks.SchedulerTask.safeCall(SchedulerTask.java:206)
at net.datenwerke.scheduler.service.scheduler.tasks.SchedulerTask.call(SchedulerTask.java:101)
at net.datenwerke.scheduler.service.scheduler.tasks.SchedulerTask.call(SchedulerTask.java:1)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)I've got a few questions:
1. How can we stop the jobs that are showing as "In Execution" via the UI? If I log in as root, I can still only see jobs by root or to root, not ALL jobs in the system
2. When I log in as root and go to the Scheduler, the "Archive" button also has an option to "Remove". When I log in as other users we've created, "Remove" is not there. How should this privilege be granted (via script)?
3. Is it normal / expected that jobs that are "In Execution" stay in that state even after ReportServer restarts?
4. If you want to schedule a report with inputted (non-default) values for the parameters, do you have to do it by first saving a variant with those values? It seems if we just enter parameter values and choose Schedule, the report runs with whatever the default values were for the report parameters.
Thanks,
Alana
Offline
A few updates on my questions:
1 - I see now that under Administration there is a Scheduler link that seems to show all scheduled jobs, not just those for the current user. I guess I could have deleted the "In Execution" jobs for another user from here, as the root user
2 - Is this related to "SchedulingBasicSecurityTarget" vs "SchedulingAdminSecurityTarget" ? I wouldn't want to let a user have access to the Administration / Scheduler section where they could access everyone's job. It would just be nice to give them the privilege to remove their own jobs (not just archive).
Offline
Hi Alana,
1: Correct; the admin Scheduler view allows users with the necessary permissions to show all scheduler jobs in the system while the standard scheduler module only show jobs where the user is directly involved (as the one who created the task, or as a recipient)
2: The delete permission is controlled by the EXECUTE right on SchedulingAdminSecurityTarget. If the permission is set a user can permanently delete jobs instead of moving them to the archive. (The READ right controls the visibility of the Scheduler section on the admin module)
3: If ReportServer restarts, jobs that were not completed before the restart get marked as failed and then are restarted automatically. Additionally there was a bug that caused jobs to get restarted even if they were already archived. I opened a ticket in the support portal for you and will attach a patch there as soon as it becomes available.
4. Correct. The scheduler has no own copy of a scheduled report but only a link. So if you want to schedule a report with on-devault values you have to create a variant.
Cheers,
Thorsten
Hi Thorsten,
1, and 2. perfect, makes sense.
3. "If ReportServer restarts, jobs that were not completed before the restart get marked as failed and then are restarted automatically" --> Do you mean jobs that were in status of "In Execution" prior to the restart should get updated to "Failed" during the shutdown process and then run again on startup? If I schedule a job to run once and it fails, it seems to re-schedule it to run again tomorrow at the same time. If I was to restart the server before that time, this job should not run until that specified time, correct?
4. Makes sense, thanks.
One more question...
5. Regarding the Scheduler: with either the report or a variant, in both cases I can leave mandatory parameters empty and still choose the Schedule option. It is only when the scheduled job tries to run that it fails and the job details show a stack trace indicating "Required parameter <param> is not set".
Would it be possible to somehow validate that mandatory parameters have been supplied at the time the job is being scheduled?
Thanks,
Alana
Offline
Hi Alana,
3: On startup all jobs that are still marked as executing (of course the actual job was cancelled when the server restarted, but the status table still has them in the executing state) are set to failed. If the job is still active (was not archived) it gets scheduled once for immediate execution.
In general there is no re-scheduling of failed jobs. If a job fails, the normal schedule is resumed. If you are using a trigger that has a fixed execution count the count only refers to successful executions, which is way you see the additional execution in your example.
5: This shouldn't be too difficult to add. I've created a ticket and we will look into it.
Cheers,
Thorsten
Hi Thorsten,
3: Thanks for the explanation - that makes sense, and yes, that also explains my scenario (I had set it to "1 execution").
So if we have a scheduled job that is in the executing state and taking too long (or we just want to stop it), what would be the appropriate way to do this? Archiving would make sure it didn't execute again if we re-started Tomcat, but would it still be continuing to run in the meantime? Would removing the job completely stop the execution?
5: Great, that would be helpful.
Thanks,
Alana
Offline
Hi Alana,
there currently is no good way to stop individual scheduler tasks - the best you can do is shutdown the entire scheduler daemon. This will in turn shut down the threadpool that is used to execute the scheduler jobs.
Shutting down the threadpool means nothing more than (after a 1 minute grace peroid) sending an interrupt to all scheduler threads. If and to what extend this really stops the execution depends on the report engine in use and even the specific step of the execution - so for example not all jdbc drivers support aborting a statement.
To stop the scheduler daemon use the "scheduler daemon stop/start" terminal command.
I will also create a ticket for us to revisit this issue, but I don't expect any short term improvements.
Cheers,
Thorsten