I didn’t take English in Year 13 (it was my weakest subject, too) and got into Harvard and Stanford, so I take it that it doesn’t matter too much.
Sidenote, while I didn’t enjoy NCEA English because I thought it was overly prescriptive, I highly recommend improving your writing ability (which high schoolers unfortunately conflate with English, where English is analysis of text (like “what did the author mean in this book”) whereas writing ability is being able to articulate/argue/convince/communicate. Since the only time we (or at least for me) write during high school is in English class, we conflate the two (very different) things.
Tl;dr writing is really important so drop English by all means, but make sure you are a good writer.
Good luck with your college apps!