The management of Returnable Transport Items (RTIs), also called Reusable Transit Packaging (RTP), within automotive Just-in-Time (JIT) supply chains presents significant operational and strategic challenges, particularly for second-tier suppliers who face high demand volatility and limited control over RTI availability. Inefficient RTI flows lead to increased costs, service failures, and adverse environmental impacts. This paper addresses the complex problem of optimizing production scheduling and reusable container logistics for a second-tier plastic injection supplier by proposing a novel hybrid approach that integrates Mixed-Integer Linear Programming (MILP) with Deep Reinforcement Learning (DRL). The MILP component models detail operational decisions, including production sequencing with mold changeovers, inventory management for parts and containers (both reusable and disposable), and explicit transshipment operations, aiming to minimize total systemic costs including an environmental penalty for CO2 emissions. The DRL agent learns an adaptive policy to strategically determine the optimal initial inventory of empty reusable containers at the beginning of each planning cycle, dynamically informing the MILP model. Comprehensive computational experiments on a variety of synthetically generated instances, characterized by diverse demand patterns (Stable, Peaks, Volatile), demonstrate the proposed hybrid approach's effectiveness. Results indicate that the MILP-DRL approach achieves competitive total system costs and significantly reduces service level failures, while effectively navigating the trade-offs between operational costs, backorders, transshipments, and CO2 emissions. The study provides valuable insights into the benefits of adaptive, learning-based strategies for RTI management and offers practical guidance for second-tier suppliers striving to enhance efficiency and sustainability in demanding JIT environments.