That's strange - when I put the ESB MKV in TS Muxer, the chapters work perfectly fine. But either way, here are the chapter times I used for ESB:

00:00:00.000
00:00:21.021
00:01:55.115
00:04:15.822
00:08:17.863
00:10:07.072
00:12:49.301
00:15:03.835
00:16:20.945
00:18:18.562
00:19:36.040
00:21:13.971
00:23:49.526
00:25:41.003
00:29:19.922
00:31:20.275
00:33:29.137
00:35:46.607
00:36:42.663
00:38:24.498
00:40:27.187
00:44:10.676
00:45:59.751
00:50:04.495
00:52:06.117
00:53:47.985
00:57:08.185
01:00:25.915
01:03:15.518
01:05:12.468
01:07:40.415
01:12:52.293
01:16:01.348
01:17:46.453
01:18:54.020
01:22:03.109
01:24:45.972
01:26:51.030
01:29:28.520
01:33:39.504
01:37:28.132
01:39:25.583
01:40:45.729
01:42:51.520
01:46:02.177
01:48:16.511
01:52:36.637
01:54:42.996
01:58:14.941
02:00:25.571

And here are ROTJ chapters (they are not tested, so you should probably check, if the chapters are in the right places before burning):

00:00:00.000
00:00:21.021
00:01:55.115
00:04:59.465
00:08:11.123
00:12:23.107
00:13:55.933
00:17:07.390
00:21:41.664
00:24:40.309
00:29:08.777
00:30:44.773
00:36:58.446
00:38:45.887
00:44:31.232
00:47:37.918
00:50:49.876
00:52:46.993
00:53:38.544
00:56:08.494
00:57:58.104
01:01:55.073
01:05:32.857
01:07:01.712
01:11:14.764
01:15:06.396
01:17:29.038
01:21:44.293
01:25:20.275
01:26:05.320
01:27:04.712
01:29:18.679
01:32:32.839
01:35:54.907
01:39:38.864
01:41:36.715
01:43:41.372
01:44:40.764
01:47:55.859
01:49:54.378
01:50:51.201
01:53:50.713
01:54:21.477
01:55:10.992
01:56:53.494
01:58:47.975
02:01:12.285
02:04:04.257
02:04:40.526
02:06:28.534