环境：虚拟机
[root@rac1 ~]# lsb_release -aLSB Version: :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch
rinting-4.0-amd64rinting-4.0-noarchDistributor ID: CentOSDescription: CentOS release 6.10 (Final)Release: 6.10Codename: Final[root@rac1 ~]# uname -r2.6.32-754.el6.x86_64SQL*Plus: Release 11.2.0.4.0 Production on Fri Feb 22 09:06:15 2019 在安装过程中，grid，asm，db软件安装都还比较顺利，但是在dbca的过程中，13%
[root@rac1 trace]# tail -n 200 alert_bol1.log
ALTER DATABASE DEFAULT TABLESPACE "USERS"Completed: ALTER DATABASE DEFAULT TABLESPACE "USERS"Fri Feb 22 12:42:13 2019Thread 1 advanced to log sequence 2 (LGWR switch) Current log# 2 seq# 2 mem# 0: +DATA/bol/onlinelog/group_2.267.1000903313 Current log# 2 seq# 2 mem# 1: +FRA/bol/onlinelog/group_2.261.1000903313Fri Feb 22 12:42:31 2019Thread 1 cannot allocate new log, sequence 3Checkpoint not complete Current log# 2 seq# 2 mem# 0: +DATA/bol/onlinelog/group_2.267.1000903313 Current log# 2 seq# 2 mem# 1: +FRA/bol/onlinelog/group_2.261.1000903313Thread 1 advanced to log sequence 3 (LGWR switch) Current log# 1 seq# 3 mem# 0: +DATA/bol/onlinelog/group_1.266.1000903311 Current log# 1 seq# 3 mem# 1: +FRA/bol/onlinelog/group_1.260.1000903311Fri Feb 22 12:43:04 2019Thread 1 cannot allocate new log, sequence 4Checkpoint not complete Current log# 1 seq# 3 mem# 0: +DATA/bol/onlinelog/group_1.266.1000903311 Current log# 1 seq# 3 mem# 1: +FRA/bol/onlinelog/group_1.260.1000903311Thread 1 advanced to log sequence 4 (LGWR switch) Current log# 2 seq# 4 mem# 0: +DATA/bol/onlinelog/group_2.267.1000903313 Current log# 2 seq# 4 mem# 1: +FRA/bol/onlinelog/group_2.261.1000903313Fri Feb 22 12:43:15 2019WARNING: Write Failed. group:3 disk:0 AU:315 offset:210432 size:1024WARNING: Write Failed. group:2 disk:1 AU:1026 offset:210432 size:1024Errors in file /u01/app/oracle/diag/rdbms/bol/bol1/trace/bol1_lgwr_13834.trc:ORA-15080: synchronous I/O operation to a disk failedORA-27061: waiting for async I/Os failedLinux-x86_64 Error: 5: Input/output errorAdditional information: -1Additional information: 1024WARNING: failed to write mirror side 1 of virtual extent 12 logical extent 0 of file 267 in group 2 on disk 1 allocation unit 1026 Errors in file /u01/app/oracle/diag/rdbms/bol/bol1/trace/bol1_lgwr_13834.trc:ORA-15080: synchronous I/O operation to a disk failedORA-27061: waiting for async I/Os failedLinux-x86_64 Error: 5: Input/output errorAdditional information: -1Additional information: 1024WARNING: failed to write mirror side 1 of virtual extent 12 logical extent 0 of file 261 in group 3 on disk 0 allocation unit 315 Errors in file /u01/app/oracle/diag/rdbms/bol/bol1/trace/bol1_lgwr_13834.trc:ORA-00345: redo log write error block 24987 count 2ORA-00312: online log 2 thread 1: '+DATA/bol/onlinelog/group_2.267.1000903313'ORA-15081: failed to submit an I/O operation to a diskORA-15081: failed to submit an I/O operation to a diskORA-00345: redo log write error block 24987 count 2ORA-00312: online log 2 thread 1: '+FRA/bol/onlinelog/group_2.261.1000903313'ORA-15081: failed to submit an I/O operation to a diskORA-15081: failed to submit an I/O operation to a diskErrors in file /u01/app/oracle/diag/rdbms/bol/bol1/trace/bol1_lgwr_13834.trc:ORA-00340: IO error processing online log 2 of thread 1ORA-00345: redo log write error block 24987 count 2ORA-00312: online log 2 thread 1: '+DATA/bol/onlinelog/group_2.267.1000903313'ORA-15081: failed to submit an I/O operation to a diskORA-15081: failed to submit an I/O operation to a diskORA-00345: redo log write error block 24987 count 2ORA-00312: online log 2 thread 1: '+FRA/bol/onlinelog/group_2.261.1000903313'ORA-15081: failed to submit an I/O operation to a diskORA-15081: failed to submit an I/O operation to a diskLGWR (ospid: 13834): terminating the instance due to error 340Fri Feb 22 12:43:16 2019System state dump requested by (instance=1, osid=13834 (LGWR)), summary=[abnormal instance termination].System State dumped to trace file /u01/app/oracle/diag/rdbms/bol/bol1/trace/bol1_diag_13812_20190222124316.trcDumping diagnostic data in directory=[cdmp_20190222124316], requested by (instance=1, osid=13834 (LGWR)), summary=[abnormal instance termination].Instance terminated by LGWR, pid = 13834
asm alert日志
Fri Feb 22 17:32:48 2019NOTE: client bol1:bol registered, osid 30819, mbr 0x1Fri Feb 22 17:32:51 2019NOTE: Advanced to new COD format for group DATANOTE: Advanced to new COD format for group FRAFri Feb 22 17:34:30 2019NOTE: ASM client bol1:bol disconnected unexpectedly.NOTE: check client alert log.NOTE: Trace records dumped in trace file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_30819.trc遇到此问题，一开始以为是缺少什么安装包或者共享磁盘的配置问题
在换一套安装依然如此
第一套使用udev，第二套使用asmlib
asm相关状态
[grid@rac2 ~]$ crs_stat -t -vName Type R/RA F/FT Target State Host ----------------------------------------------------------------------ora.DATA.dg ora....up.type 0/5 0/ ONLINE ONLINE rac1 ora.FRA.dg ora....up.type 0/5 0/ ONLINE ONLINE rac1 ora....ER.lsnr ora....er.type 0/5 0/ ONLINE ONLINE rac1 ora....N1.lsnr ora....er.type 0/5 0/0 ONLINE ONLINE rac1 ora.OCR.dg ora....up.type 0/5 0/ ONLINE ONLINE rac1 ora.asm ora.asm.type 0/5 0/ ONLINE ONLINE rac1 ora.cvu ora.cvu.type 0/5 0/0 ONLINE ONLINE rac1 ora.gsd ora.gsd.type 0/5 0/ OFFLINE OFFLINE ora....network ora....rk.type 0/5 0/ ONLINE ONLINE rac1 ora.oc4j ora.oc4j.type 0/1 0/2 ONLINE ONLINE rac1 ora.ons ora.ons.type 0/3 0/ ONLINE ONLINE rac1 ora....SM1.asm application 0/5 0/0 ONLINE ONLINE rac1 ora....C1.lsnr application 0/5 0/0 ONLINE ONLINE rac1 ora.rac1.gsd application 0/5 0/0 OFFLINE OFFLINE ora.rac1.ons application 0/3 0/0 ONLINE ONLINE rac1 ora.rac1.vip ora....t1.type 0/0 0/0 ONLINE ONLINE rac1 ora....SM2.asm application 0/5 0/0 ONLINE ONLINE rac2 ora....C2.lsnr application 0/5 0/0 ONLINE ONLINE rac2 ora.rac2.gsd application 0/5 0/0 OFFLINE OFFLINE ora.rac2.ons application 0/3 0/0 ONLINE ONLINE rac2 ora.rac2.vip ora....t1.type 0/0 0/0 ONLINE ONLINE rac2 ora.scan1.vip ora....ip.type 0/0 0/0 ONLINE ONLINE rac1
[grid@rac1 .ssh]$ olsnodes -nrac1 1rac2 2[grid@rac1 .ssh]$ olsnodes -n -i -s -trac1 1 rac1vip Active Unpinnedrac2 2 rac2vip Active Unpinned[grid@rac1 .ssh]$ srvctl status asm -aASM is running on rac2,rac1ASM is enabled.
[grid@rac2 ~]$ crsctl query css votedisk## STATE File Universal Id File Name Disk group-- ----- ----------------- --------- --------- 1. ONLINE 4c22914de7394f7abfd6c9635cb1c1fd (ORCL:OCR_VOTE1) [OCR] 2. ONLINE 0f6aab9aaa744faebfc22d0622804518 (ORCL:OCR_VOTE2) [OCR] 3. ONLINE 1d1fd2ecffcf4f76bf0ea356730a9c52 (ORCL:OCR_VOTE3) [OCR]Located 3 voting disk(s).
[grid@rac1 trace]$ ps -ef|grep lsnr|grep -v 'grep'|grep -v 'ocfs'|awk '{print$9}'LISTENERLISTENER_SCAN1 [grid@rac1 ~]$ asmcmdASMCMD> lsdgState Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files NameMOUNTED EXTERN N 512 4096 1048576 163840 161875 0 161875 0 N DATA/MOUNTED EXTERN N 512 4096 1048576 81920 81692 0 81692 0 N FRA/MOUNTED EXTERN N 512 4096 1048576 6144 5744 0 5744 0 Y OCR/
在查找过程中，找到相似的文档和案例
听说这次用的存储是自定义？ 第一次使用的。。。目前还没有搞定此问题
--添加详细日志
[root@rac1 ~]# ls -l /dev/asm*brw-rw---- 1 grid asmadmin 8, 16 Feb 26 09:00 /dev/asm-diskbbrw-rw---- 1 grid asmadmin 8, 32 Feb 25 18:30 /dev/asm-diskcbrw-rw---- 1 grid asmadmin 8, 48 Feb 25 21:31 /dev/asm-diskdbrw-rw---- 1 grid asmadmin 8, 64 Feb 25 23:11 /dev/asm-diskebrw-rw---- 1 grid asmadmin 8, 80 Feb 25 18:22 /dev/asm-diskfbrw-rw---- 1 grid asmadmin 8, 96 Feb 25 23:11 /dev/asm-diskg[root@rac1 dev]# ll /dev/sd*brw-rw---- 1 root disk 8, 0 Feb 25 16:46 /dev/sdabrw-rw---- 1 root disk 8, 1 Feb 25 16:46 /dev/sda1brw-rw---- 1 root disk 8, 2 Feb 25 16:46 /dev/sda2brw-rw---- 1 root disk 8, 16 Feb 25 16:59 /dev/sdbbrw-rw---- 1 root disk 8, 32 Feb 25 16:59 /dev/sdcbrw-rw---- 1 root disk 8, 48 Feb 25 16:59 /dev/sddbrw-rw---- 1 root disk 8, 64 Feb 25 16:59 /dev/sdebrw-rw---- 1 root disk 8, 80 Feb 25 16:59 /dev/sdfbrw-rw---- 1 root disk 8, 96 Feb 25 16:59 /dev/sdg[root@rac1 ~]# ll /u01/app/oracle/total 20drwxr-x--- 3 oracle oinstall 4096 Feb 25 17:58 admindrwxr-x--- 4 oracle oinstall 4096 Feb 25 17:55 cfgtoollogsdrwxr-xr-x 2 oracle oinstall 4096 Feb 25 17:52 checkpointsdrwxrwxr-x 11 oracle oinstall 4096 Feb 25 17:45 diagdrwxr-xr-x 3 oracle oinstall 4096 Feb 25 17:40 product[grid@rac1 ~]$ crs_stat -t -vCRS-0184: Cannot communicate with the CRS daemon.[root@rac1 ~]# /u01/app/11.2.0/grid/bin/crsctl start crsCRS-4640: Oracle High Availability Services is already activeCRS-4000: Command Start failed, or completed with errors.SQL> conn /as sysasm Connected.SQL> select name,state from v$asm_diskgroup;NAME STATE------------------------------ -----------OCR DISMOUNTEDFRA DISMOUNTEDDATA DISMOUNTEDSQL> ALTER DISKGROUP ALL MOUNT;[root@rac1 ~]# tail -f -n 300 /u01/app/grid/diag/asm/+asm/+ASM1/trace/alert_+ASM1.log ORA-15078: ASM diskgroup was forcibly dismountedErrors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_14673.trc:ORA-15078: ASM diskgroup was forcibly dismountedErrors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_14673.trc:ORA-15078: ASM diskgroup was forcibly dismountedMon Feb 25 23:16:52 2019NOTE: ASMB process exiting due to lack of ASM file activity for 305 secondsMon Feb 25 23:16:52 2019NOTE: client +ASM1:+ASM deregisteredMon Feb 25 23:18:24 2019Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_14673.trc:ORA-15078: ASM diskgroup was forcibly dismountedErrors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_14673.trc:Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_14673.trc:ORA-15078: ASM diskgroup was forcibly dismountedWARNING: requested mirror side 1 of virtual extent 4 logical extent 0 offset 446464 is not allocated; I/O request failedErrors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_14673.trc:ORA-15078: ASM diskgroup was forcibly dismountedMon Feb 25 23:18:24 2019* allocate domain 3, invalid = TRUE kjbdomatt send to inst 2NOTE: attached to recovery domain 3NOTE: starting recovery of thread=2 ckpt=2.0 group=3 (OCR)NOTE: advancing ckpt for group 3 (OCR) thread=2 ckpt=2.0NOTE: cache recovered group 3 to fcn 0.531NOTE: redo buffer size is 256 blocks (1053184 bytes)NOTE: LGWR attempting to mount thread 1 for diskgroup 3 (OCR)NOTE: LGWR found thread 1 closed at ABA 3.36NOTE: LGWR mounted thread 1 for diskgroup 3 (OCR)NOTE: LGWR opening thread 1 at fcn 0.531 ABA 4.37NOTE: cache mounting group 3/0xB2FAEF18 (OCR) succeededNOTE: cache ending mount (success) of group OCR number=3 incarn=0xb2faef18Tue Feb 26 09:07:00 2019NOTE: Instance updated compatible.asm to 11.2.0.0.0 for grp 1SUCCESS: diskgroup DATA was mountedNOTE: Instance updated compatible.asm to 11.2.0.0.0 for grp 2SUCCESS: diskgroup FRA was mountedNOTE: Instance updated compatible.asm to 11.2.0.0.0 for grp 3SUCCESS: diskgroup OCR was mountedSUCCESS: ALTER DISKGROUP ALL MOUNTNOTE: Attempting voting file refresh on diskgroup OCRNOTE: Refresh completed on diskgroup OCR. Found 1 voting file(s).NOTE: Voting file relocation is required in diskgroup OCRNOTE: Attempting voting file relocation on diskgroup OCRNOTE: Successful voting file relocation on diskgroup OCRTue Feb 26 09:07:02 2019WARNING: failed to online diskgroup resource ora.DATA.dg (unable to communicate with CRSD/OHASD)WARNING: failed to online diskgroup resource ora.FRA.dg (unable to communicate with CRSD/OHASD)WARNING: failed to online diskgroup resource ora.OCR.dg (unable to communicate with CRSD/OHASD)[root@rac1 ~]# vim /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_14673.trcTrace file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_14673.trcOracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit ProductionWith the Real Application Clusters and Automatic Storage Management optionsORACLE_HOME = /u01/app/11.2.0/gridSystem name: LinuxNode name: rac1Release: 2.6.32-754.el6.x86_64Version: #1 SMP Tue Jun 19 21:26:04 UTC 2018Machine: x86_64VM name: VMWare Version: 6Instance name: +ASM1Redo thread mounted by this instance: 0Oracle process number: 24Unix process pid: 14673, image: oracle@rac1 (TNS V1-V3)*** 2019-02-25 17:17:34.002*** SESSION ID:(13.3) 2019-02-25 17:17:34.002*** CLIENT ID:() 2019-02-25 17:17:34.002*** SERVICE NAME:() 2019-02-25 17:17:34.002*** MODULE NAME:(crsd.bin@rac1 (TNS V1-V3)) 2019-02-25 17:17:34.002*** ACTION NAME:() 2019-02-25 17:17:34.002WARNING: Write Failed. group:1 disk:2 AU:23 offset:151552 size:4096path:/dev/asm-diskd incarnation:0xe96a1e30 synchronous result:'I/O error' subsys:System iop:0x7f688dfdd000 bufp:0x7f688e3d4a00 osderr:0x0 osderr1:0x0ORA-15080: synchronous I/O operation to a disk failedORA-27072: File I/O errorLinux-x86_64 Error: 5: Input/output errorAdditional information: 4Additional information: 47400Additional information: -1WARNING: failed to write mirror side 1 of virtual extent 0 logical extent 0 of file 255 in group 1 on disk 2 allocation unit 23ORA-15081: failed to submit an I/O operation to a diskORA-27072: File I/O errorLinux-x86_64 Error: 5: Input/output errorAdditional information: 4Additional information: 47400Additional information: -1WARNING: Write Failed. group:1 disk:2 AU:23 offset:118784 size:4096path:/dev/asm-diskd incarnation:0xe96a1e30 synchronous result:'I/O error' subsys:System iop:0x7f688dfdd000 bufp:0x7f688e3d4a00 osderr:0x0 osderr1:0x0ORA-15080: synchronous I/O operation to a disk failedORA-27072: File I/O errorLinux-x86_64 Error: 5: Input/output errorAdditional information: 4Additional information: 47400Additional information: -1WARNING: failed to write mirror side 1 of virtual extent 0 logical extent 0 of file 255 in group 1 on disk 2 allocation unit 23ORA-15081: failed to submit an I/O operation to a diskORA-27072: File I/O errorLinux-x86_64 Error: 5: Input/output errorAdditional information: 4Additional information: 47400Additional information: -1WARNING: Write Failed. group:1 disk:2 AU:23 offset:118784 size:4096path:/dev/asm-diskd incarnation:0xe96a1e30 synchronous result:'I/O error' subsys:System iop:0x7f688dfdd000 bufp:0x7f688e3d4a00 osderr:0x0 osderr1:0x0ORA-15080: synchronous I/O operation to a disk failedORA-27072: File I/O errorLinux-x86_64 Error: 5: Input/output errorAdditional information: 4Additional information: 47336Additional information: -1WARNING: failed to write mirror side 1 of virtual extent 0 logical extent 0 of file 255 in group 1 on disk 2 allocation unit 23ORA-15081: failed to submit an I/O operation to a diskORA-27072: File I/O errorLinux-x86_64 Error: 5: Input/output errorAdditional information: 4Additional information: 47336Additional information: -1*** 2019-02-25 17:28:27.478Received ORADEBUG command (#1) 'CLEANUP_KFK_FD' from process 'Unix process pid: 18224, image: '*** 2019-02-25 17:28:27.478Finished processing ORADEBUG command (#1) 'CLEANUP_KFK_FD'*** 2019-02-25 23:11:47.745WARNING:failed xlate 1ORA-15078: ASM diskgroup was forcibly dismountedWARNING:failed xlate 1*** 2019-02-25 23:11:48.098WARNING:failed xlate 1ORA-15078: ASM diskgroup was forcibly dismountedWARNING:failed xlate 1ORA-15078: ASM diskgroup was forcibly dismounted*** 2019-02-25 23:11:58.466Received ORADEBUG command (#2) 'CLEANUP_KFK_FD' from process 'Unix process pid: 29490, image: '*** 2019-02-25 23:11:58.467Finished processing ORADEBUG command (#2) 'CLEANUP_KFK_FD'*** 2019-02-25 23:12:13.475Received ORADEBUG command (#3) 'CLEANUP_KFK_FD' from process 'Unix process pid: 29490, image: '*** 2019-02-25 23:12:13.475Finished processing ORADEBUG command (#3) 'CLEANUP_KFK_FD'ORA-15078: ASM diskgroup was forcibly dismountedksfdrfms:Mirror Read file=+OCR.255.4294967295 fob=0xa0802c18 bufp=0x7f688e3d4a00 blkno=1133 nbytes=4096WARNING:failed xlate 1WARNING: requested mirror side 1 of virtual extent 4 logical extent 0 offset 446464 is not allocated; I/O request failedksfdrfms:Read failed from mirror side=1 logical extent number=0 dskno=65535ORA-15078: ASM diskgroup was forcibly dismounted[root@rac1 log]# tail -n 300 /var/log/messagesFeb 25 16:47:34 rac1 rtkit-daemon[2828]: Sucessfully made thread 2826 of process 2826 (/usr/bin/pulseaudio) owned by '42' high priority at nice level -11.Feb 25 17:13:36 rac1 root: exec /u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib /u01/app/11.2.0/grid/bin/crswrapexece.pl /u01/app/11.2.0/grid/crs/install/s_crsconfig_rac1_env.txt /u01/app/11.2.0/grid/bin/ohasd.bin "reboot"Feb 25 17:17:33 rac1 kernel: sd 3:0:0:0: reservation conflictFeb 25 17:17:33 rac1 kernel: sd 3:0:0:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_OKFeb 25 17:17:33 rac1 kernel: sd 3:0:0:0: [sdb] CDB: Read(10): 28 00 00 04 00 14 00 00 01 00Feb 25 17:17:33 rac1 kernel: end_request: critical nexus error, dev sdb, sector 262164Feb 25 17:17:34 rac1 kernel: sd 3:0:2:0: reservation conflictFeb 25 17:17:34 rac1 kernel: sd 3:0:2:0: [sdd] Result: hostbyte=DID_OK driverbyte=DRIVER_OKFeb 25 17:17:34 rac1 kernel: sd 3:0:2:0: [sdd] CDB: Write(10): 2a 00 00 00 b9 28 00 00 08 00Feb 25 17:17:34 rac1 kernel: end_request: critical nexus error, dev sdd, sector 47400....Feb 25 23:49:02 rac1 kernel: end_request: critical nexus error, dev sdb, sector 262148Feb 26 06:10:49 rac1 kernel: sd 3:0:0:0: reservation conflictFeb 26 06:10:49 rac1 kernel: sd 3:0:0:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_OKFeb 26 06:10:49 rac1 kernel: sd 3:0:0:0: [sdb] CDB: Read(10): 28 00 00 04 00 03 00 00 01 00Feb 26 06:10:49 rac1 kernel: end_request: critical nexus error, dev sdb, sector 262147Feb 26 06:38:40 rac1 kernel: sd 3:0:0:0: reservation conflictFeb 26 06:38:40 rac1 kernel: sd 3:0:0:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_OKFeb 26 06:38:40 rac1 kernel: sd 3:0:0:0: [sdb] CDB: Read(10): 28 00 00 04 00 03 00 00 01 00Feb 26 06:38:40 rac1 kernel: end_request: critical nexus error, dev sdb, sector 262147Feb 26 08:39:13 rac1 kernel: sd 3:0:0:0: reservation conflictFeb 26 08:39:13 rac1 kernel: sd 3:0:0:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_OKFeb 26 08:39:13 rac1 kernel: sd 3:0:0:0: [sdb] CDB: Read(10): 28 00 00 04 00 04 00 00 01 00Feb 26 08:39:13 rac1 kernel: end_request: critical nexus error, dev sdb, sector 262148[root@rac1 ~]# tail -n 200 /u01/app/oracle/diag/rdbms/bol/bol1/trace/alert_bol1.log ALTER DATABASE DEFAULT TABLESPACE "USERS"Completed: ALTER DATABASE DEFAULT TABLESPACE "USERS"Mon Feb 25 18:22:32 2019Thread 1 advanced to log sequence 2 (LGWR switch) Current log# 2 seq# 2 mem# 0: +DATA/bol/onlinelog/group_2.267.1001182931 Current log# 2 seq# 2 mem# 1: +FRA/bol/onlinelog/group_2.261.1001182931Mon Feb 25 18:22:49 2019WARNING: Write Failed. group:3 disk:0 AU:333 offset:402432 size:1536WARNING: Write Failed. group:2 disk:1 AU:1136 offset:402432 size:1536Errors in file /u01/app/oracle/diag/rdbms/bol/bol1/trace/bol1_lgwr_32760.trc:ORA-15080: synchronous I/O operation to a disk failedORA-27061: waiting for async I/Os failedLinux-x86_64 Error: 5: Input/output errorAdditional information: -1Additional information: 1536WARNING: failed to write mirror side 1 of virtual extent 30 logical extent 0 of file 267 in group 2 on disk 1 allocation unit 1136 Errors in file /u01/app/oracle/diag/rdbms/bol/bol1/trace/bol1_lgwr_32760.trc:ORA-15080: synchronous I/O operation to a disk failedORA-27061: waiting for async I/Os failedLinux-x86_64 Error: 5: Input/output errorAdditional information: -1Additional information: 1536WARNING: failed to write mirror side 1 of virtual extent 30 logical extent 0 of file 261 in group 3 on disk 0 allocation unit 333 Errors in file /u01/app/oracle/diag/rdbms/bol/bol1/trace/bol1_lgwr_32760.trc:ORA-00345: redo log write error block 62226 count 3ORA-00312: online log 2 thread 1: '+DATA/bol/onlinelog/group_2.267.1001182931'ORA-15081: failed to submit an I/O operation to a diskORA-15081: failed to submit an I/O operation to a diskORA-00345: redo log write error block 62226 count 3ORA-00312: online log 2 thread 1: '+FRA/bol/onlinelog/group_2.261.1001182931'ORA-15081: failed to submit an I/O operation to a diskORA-15081: failed to submit an I/O operation to a diskErrors in file /u01/app/oracle/diag/rdbms/bol/bol1/trace/bol1_lgwr_32760.trc:ORA-00340: IO error processing online log 2 of thread 1ORA-00345: redo log write error block 62226 count 3ORA-00312: online log 2 thread 1: '+DATA/bol/onlinelog/group_2.267.1001182931'ORA-15081: failed to submit an I/O operation to a diskORA-15081: failed to submit an I/O operation to a diskORA-00345: redo log write error block 62226 count 3ORA-00312: online log 2 thread 1: '+FRA/bol/onlinelog/group_2.261.1001182931'ORA-15081: failed to submit an I/O operation to a diskORA-15081: failed to submit an I/O operation to a diskLGWR (ospid: 32760): terminating the instance due to error 340Mon Feb 25 18:22:49 2019System state dump requested by (instance=1, osid=32760 (LGWR)), summary=[abnormal instance termination].System State dumped to trace file /u01/app/oracle/diag/rdbms/bol/bol1/trace/bol1_diag_32738_20190225182249.trcDumping diagnostic data in directory=[cdmp_20190