Coupang 에서 Kotlin 을 적용하여 개발한 사례

1. Kotlin Backend @ Coupang
debop@coupang.com

2. Who is @debop
• Since 1993
• Robotics, BPM Solution, Healthcare
• C/C++, Object Pascal, C#, Java, Scala, Kotlin
• Use Kotlin since 2016.08
• Sr. Principle Software Engineer in Coupang (2017~)
• 남의 코드 뒤집기 / 빨간펜 선생
• Common Library 제작
• Architecture 참견

3. Agenda
• Kotlin 도입 동기
• Kotlin 도입 사례
• Kotlinx (Common Library)
• 공용 Library 개발
• spring-data-requery
• Lesson & Learn

4. Kotlin 도입 동기

5. What is our problem?
• 개발자 수준 편차가 크다
• 조직 수준은 최하위 개발자 수준의 의해 결정된다

6. Our status
• Lack of Safety code
• Not Readable code
• Lack of Testing (Unit & Integration)
• Heavy oriented to JPA
• Lack of experience to Async/Non-Blocking
Programming

7. Our environments
• 1일 데이터 처리량 : 천만건(2016) -> 10억건(2018)
• MySQL (Aurora) : Replication delay120 sec (max)
• Backend Stack in 2016
• MySQL, RabbitMQ, Hive
• Spring 3, JPA

8. Our Challenges
• Akka Cluster with Lagom Framework (CQRS)
• Move to NoSQL (Cassandra, HBase)
• Migrate RabbitMQ to Kafka
• Adopting Scala (Pilot)
• Akka, Spark …

9. Failure
• Adopting Scala
• Learning Curve (Almost Junior)
• Lack of Tech Leaders
• Try to develop too complex system
• Adopting Kafka
• Simultaneous development (duplicate same error)
• Offset managements

10. To pursue change
• 객관적 시각 유지
• History 및 현실 상황 대한 이해 (기술부채)
• 현 조직의 개발 역량 객관적 평가
• 동기부여 - 필요성 설득보다는 자각할 수 있도록 자극
• 충분한 학습 시간
• 변화 경험 공유
• ASP -> C#, Java -> Scala

11. Prepare
• Code Quality 중요성 자각 -> Code Review
• Enterprise Application Commons
• Patterns 교육, Library 제공
• Enterprise Architecture 학습
• Asynchronous Programming 학습
• Kotlin Language 학습
• Kotlin is language for Android ?

12. Adopting Kotlin
• Safety Code / Readable Code
• Simple grammar (Easy to learn)
• Interoperability between Java and Kotlin
• Extension methods
• Asynchronous Programming by Coroutines

13. Environments 준비
• 개발 Tool 지원 - IntelliJ IDEA
• Static code analysis - ktlint, detekt
• Test code coverage - jacoco (sample)
• Sonarcube - sonar-kotlin plugin

14. Kotlin Backend 도입 사례

15. Use cases 2017
• Common Library (2017. 6 ~)
• Kotlin extension library
• Components (2017.08 ~ 2017.12)
• Aho-corasick, Korean Tokenizer, Kafka Client
• Standalone Web Application (2017.10~~2017.12)
• Audit Tool
• Large scale system (2017.09~2018.01)
• Catalog creation pipeline system

16. 1. Kotlin Extension library
• Coupang Standard Framework 대체
• Kotlin Best practices 제공
• Object / IO / Utils / Cache / JDBC / Spring …
• Kotlin 학습 예제 제공 (Test cases / Examples)
• Based debop4k (personal project)

17. Kotlinx-Units
• 도량형(度量衡)(Standard Unit)
• Represent Meter Units
• Weight, Length, Area, Volume, Storage Volume
• Yard-Pound Units
• Pound, Yard/Miles, Volume, Fluid Volume …
• Others
• Temperature, Electlic powers, Frequency, Light,
Radiation, Pressure …

18. Our usages
enum class Unit(val description: String, val unitConversion: BigDecimal) {
NONE("없음", BigDecimal.ZERO),
// Length
MM("mm", 0.1.toBigDecimal()),
CM("cm", BigDecimal.ONE),
M("m", 100.toBigDecimal()),
KM("km", 100_000.toBigDecimal()),

19. Our usages
// Length (Inch)
IN("in", BigDecimal.ONE),
FT("ft", 12.toBigDecimal()),
YD("yd", 36.toBigDecimal()),
// Weight (gram)
MG("mg", 0.001.toBigDecimal()),
G("g", BigDecimal.ONE),
KG("kg", 1000.toBigDecimal()),
T("t", 1_000_000.toBigDecimal()),
OZ("oz", 28.349523.toBigDecimal()),
LB("lb", 453.59237.toBigDecimal()),
•Define Units by Enum
•Not support conversions
•Not support various units
•Not support operations
•Not readable code

20. Features for units
• Readable code like this
• 5.kg() + 400.gram() => 5.4.kg()
• 100.cm() + 2.m() => 2.1.cm()
• Convert various units
• 5.kg().inGram() -> 5000.gram
• 2.5.Kb().inBytes() -> 2500.bytes()
• Convert SI Units <-> US Units

21. Use Kotlin extension methods
fun Double.millimeter(): Length = lengthOf(this, MILLIMETER)
fun Double.centimeter(): Length = lengthOf(this, CENTIMETER)
fun Double.meter(): Length = lengthOf(this)
fun Double.kilometer(): Length = lengthOf(this, KILOMETER)

22. 1. Define Unit factors
enum class LengthUnit(val unitName: String, val factor: Double) {
MILLIMETER("mm", 1.0),
CENTIMETER("cm", 10.0),
METER("m", 1e3),
KILOMETER("km", 1e6);
}

23. Define Unit class
data class Length(val value: Double = 0.0): Comparable<Length>, Serializable {
operator fun plus(other: Length): Length = Length(value + other.value)
operator fun minus(other: Length): Length = Length(value - other.value)
operator fun times(scalar: Int): Length = Length(value * scalar)
operator fun times(scalar: Long): Length = Length(value * scalar)
operator fun times(scalar: Double): Length = Length(value * scalar)
operator fun times(other: Length): Area = Area(value * other.value)
operator fun div(scalar: Int): Length = Length(value / scalar)
operator fun div(scalar: Long): Length = Length(value / scalar)
operator fun div(scalar: Double): Length = Length(value / scalar)
operator fun unaryMinus(): Length = Length(-value)
Override Operators

24. 2. Korean Tokenizer
• 중복상품 Merge 위한 Tokenizer 필요 (명사 위주)
• Twitter 에서 개발한 open-korean-text 를 Customizing
• Scala vs Kotlin 성능 비교
• Kotlin version is 1.5 ~ 3X faster with Coroutines
• 효과
• Full Indexing per Day 부하 감소 : 30%
• Elastic Search 질의 부하 : 80%

25. “제테스 MSW-3928PR 남성 비치트렁크, 95(L), 블랙블랙계열”
은전한닢 Twitter
블랙
비치
비치트렁크
계열
제테스
3928PR
블랙블랙계열
PR
MSW
,
(
)-
트렁크
3928
95
제테스
남성

26. “제테스 MSW-3928PR 남성 비치트렁크, 95(L), 블랙블랙계열”
은전한닢 Twitter
블랙
비치
비치트렁크
계열
제테스
3928PR
블랙블랙계열
PR
MSW
,
(
)-
트렁크
3928
95
제테스
남성

27. “제테스 MSW-3928PR 남성 비치트렁크, 95(L), 블랙블랙계열”
은전한닢 Twitter
블랙
비치
비치트렁크
계열
제테스
3928PR
블랙블랙계열
PR
MSW
,
(
)
L -
트렁크
3928
95
제테스
남성

28. Tokenizer Benchmark
은전한닢 Twitter RATIO
RPS 73.2 429.5 5.87 X
Avg Latency 626 ms 106 ms 5.91 X
Avg CPU Load 90% 55% 35 % Down
Twitter Tokenizer 는 한음절을 분석하지 못하는 단점이 있다

29. Scala Kotlin RATIO
Tokenize 668.620 ms 197.632 ms 3.38 X
Phrase extractor 644.902 ms 212.500 ms 3.13 X
구문 : “동해물과 백두산이 마르고 닳도록“
Benchmark by jmh
Linux Mint 18.2 Intel I7, 32GB, SSD

30. Why Kotlin is faster?
• Scala 의 loop는 느리다 - 아주 느릴 수 있다
• eclipse-collections 사용
• 메모리 절약
• Primitive type collection 지원
• Kotlin Coroutines
• 비동기 / Non-Blocking

31. 3. Kafka Client - Wilson
• 동기
• 안정된 Kafka 사용을 위해 Offset 관리가 필수
• 각 팀별 중복된 Client 구현 및 삽질
• 효과
• Message 중복 / 유실 없음 (Latest strategy)
• Retry / Circuit breaker 지원
• Metrics 를 활용한 Ack 지원
• 전사 확대 적용 중

32. Wilson message flows
Producer ConsumerKafka
Retry
Circuit Breaker
Metrics
Retry
Circuit Breaker
Metrics
Dead letters
Sending box
Redis
MySQL
Couchbase
Dead letters
Received box
Last sent timestamp Kafka Offset Manager
Message Managements
• Metrics
• Recovery / Retry
• Deadletter handling

33. Wilson Dashboard

34. 4. Audit Tool
• 상품 정보 Audit System
• developers : 1 senior, 2 junior developer
• Software stack
• React
• Spring Boot 1.5 on Vitamin Framework
• jOOQ (향후 requery로 변환 예정)
• Pilot 로 시작, 개발자들의 노력으로 정식 시스템으로 승격

35. 4. Audit Tool
Kotlin Coroutines + Spring MVC

36. 5. Creation Pipeline System
• 상품 정보 생성 프로세스 관리 시스템
• Features
• Workflow (Heavy use Kafka)
• Asynchronous / Non-blocking System
• Need High throughput

37. Seller
Retail
Kafka
Deduplication
Refining
Processing
Creation
Logging
Message Dispatcher
Creation Pipeline flow
Kafka

38. 5. Creation Pipeline System
Spring Boot 1.x Kafka 0.10.x
Kotlin 1.2.x on JVM 8
Couchbase 4.x
Zookeeper
3.x

39. %

40. %

41. Use cases 2018
• Tagging system (2018.1 ~ 2018.3)
• Spring Data Requery (alternative to JPA) (2018.4 ~)
• Rule engine (Thanos rule engine) (2018.6 ~)
• Wakanda system (Reactive + Coroutines) (2018.9 ~)
• Creation Pipeline Upgrade (2018.10 ~)

43. Aurora Cassandra ElasticSearch HBase
JVM 8
Spring DataRequery JPA/Hibernate
Virtualization
kubernetes
Docker
Spring Framework 5.xKodein
Kotlin Coroutines with Reactor / RxJava
Services
Kafka
Redis
Spring Boot 2.x Webflux with Netty / gRPC
Common Backend Stack (2018)

44. Boo1vsBoo2Performance

45. Spring MVC + Cassandra
Spring WebFlux + Cassandra Reactive
출처: Reactive Java Performance Comparison

46. 6. Tagging System
• 상품 정보에서 원하는 특성을 추출하는 시스템
• Matching 과 Refinement 에서 따로 관리
• 범용화 필요 (Configuration, Plugin)
• 기존 Java & Scala 혼용
• 장애 대응에 문제
• High Throughput

47. Tagging for Matching
스톤 9.0cm 여성하이힐 MZ-135 블랙 245
Brand Gender
Shoes
Type
Color Size

48. 7. spring-data-requery (2018)
• RequeryOperations
• Wrap EntityDataStore
• RequeryTransactionManager for TransactionManager
• Support Spring @Transactional
• Better performance than spring-data-jpa
• when exists, paging, not load all entities
https://www.slideshare.net/debop/alternatives-of-jpahibernate
https://www.slideshare.net/debop/requery-overview
https://www.slideshare.net/debop/spring-data-requery

49. spring-data-requery
• Repository built in SQL
• ByPropertyName Auto generation methods
• @Query for Native SQL Query
• Query By Example
• Not Supported
• Association Path (not specified join method)
• Named parameter in @Query (just use `?`)

50. Define Entity - Java
@Getter
@Entity(name = "BasicUser", copyable = true)
@Table(name = "basic_user")
public abstract class AbstractBasicUser extends AuditableLongEntity {
@Key
@Generated
protected Long id;
protected String name;
protected String email;
protected LocalDate birthday;
protected Integer age;
@ForeignKey
@OneToOne
protected AbstractBasicLocation address;
@ManyToMany(mappedBy = "members")
protected Set<AbstractBasicGroup> groups;
@Column(unique = true)
protected UUID uuid;

51. Use @Query in Repository
interface DeclaredQueryRepository extends RequeryRepository<BasicUser, Long> {
@Query("select * from basic_user u where u.email = ?")
BasicUser findByAnnotatedQuery(String email);
@Query("select * from basic_user u where u.email like ?")
List<BasicUser> findAllByEmailMatches(String email);
@Query("select * from basic_user u limit ?")
List<BasicUser> findWithLimits(int limit);
@Query("select * from basic_user u where u.name=? and u.email=? limit 1")
BasicUser findAllBy(String name, String email);
@Query("select u.id, u.name from basic_user u where u.email=?")
List<Tuple> findAllIds(String email);
@Query("select * from basic_user u where u.birthday = ?")
List<BasicUser> findByBirthday(LocalDate birthday);
}

52. Query By Example
BasicUser user = RandomData.randomUser();
user.setName("example");
requeryTemplate.insert(user);
BasicUser exampleUser = new BasicUser();
exampleUser.setName("EXA");
ExampleMatcher matcher = matching()
.withMatcher("name", startsWith().ignoreCase())
.withIgnoreNullValues();
Example<BasicUser> example = Example.of(exampleUser, matcher);
Return<? extends Result<BasicUser>> query = buildQueryByExample(example);
BasicUser foundUser = query.get().firstOrNull();
assertThat(foundUser).isNotNull().isEqualTo(user);

53. Query by Property
List<User> findByFirstnameOrLastname(String firstname, String lastname);
List<User> findByLastnameLikeOrderByFirstnameDesc(String lastname);
List<User> findByLastnameNotLike(String lastname);
List<User> findByLastnameNot(String lastname);
List<User> findByManagerLastname(String name);
List<User> findByColleaguesLastname(String lastname);
List<User> findByLastnameNotNull();
@Query("select u.lastname from SD_User u group by u.lastname")
Page<String> findByLastnameGrouped(Pageable pageable);
long countByLastname(String lastname);
int countUsersByFirstname(String firstname);
boolean existsByLastname(String lastname);
Note: Association Path is not supported
Note: Association Path is not supported

54. Future works
• Coroutines for Async JDBC Operations (working)
• Support Named parameter
• Support `@Param` in spring data
• Requery for SQL on Hadoop
• Apache Phoenix (HBase) (Done)
• Apache Hive, Apache Drill …

55. 8. Rule Engine (2018)
• Embedded
• Support Language
• MVEL (almost like Java spec)
• KotlinScript
• Javascript
• Support Multi Rule combination
• Rule Definition DSL (Kotlin)
• Rule Editor in WEB (in construction)

56. Rule Engine
A rule engine can be viewed as a sophisticated interpreter for if/then
statements, where the statements themselves are known as rules.

57. Rule Definition
val rule = rule {
name = "myRule"
description = "myDescription"
priority = 3
condition {
condition1.evaluate(it)
}
action {
action1.execute(it)
}
action {
action2.execute(it)
}
}
val rule = RuleBuilder()
.name("myRule")
.description("myDescription")
.priority(3)
.whenever(condition)
.then(action1)
.then(action2)
.build()
@Rule(name = "weather rule", description = "if it rains then take an umbrella")
class WeatherRule {
@Condition
fun itRains(@Fact("rain") rain: Boolean): Boolean = rain
@Action
fun takeAnUmbrella() {
println("It rains, take an umbrella")
}
}

58. 9. Wakanda
• Catalog 정보 관리 시스템의 핵심
• Read, Write Side 분리 제공
• 전사 통합에 의해 Write side만 존치
• Fully Reactive / Non-Blocking

59. Editable Catalog Process

60. Cassandra ElasticSearch
JVM 8
Cassandra Vanilla Driver
Virtualization
kubernetes
Docker
Spring Framework 5.x
Kotlin Coroutines with Reactor
Services
Kafka
Redis
Spring Boot 2.x Webflux
Wakanda Software stack

61. Lesson & Learns

62. 반성해야 하는 점
• 기본기 학습 - 닥치고 코딩 (X)
• Java와 차이점 및 Kotlin best practices 검토 필요
• 첫 술에 배부를 수 없다
• 실망하지 말라, Refactoring 은 필수다
• Coroutines 에 대한 학습 및 테스트
• 어차피 비동기는 어렵다.
• Coroutines는 하나의 방법일 뿐이다

63. 안정적 도입을 위한 Tips
• 충분한 학습 기회 & 실습
• Code Quality 중요성 인식
• Upsource 전사 활용 중 (문화가 중요)
• 강력한 동기 부여
• Tech Leader의 지속적인 Leading & Coach
• 성공 사례 만들기 (작은 것부터)

64. 효과
• Safety Code
• Readable Code
• 성능 향상
• Latency 감소, High Throughput 달성
• Asynchronous/Non-Blocking 적용 용이
• 유지보수성 향상 (장애 감소)

65. 언어별 Catalog Tribe 개발자 비율
(2017)
5%
Kotlin
10%
Scala
15%
Java
70%
Java Scala Kotlin Python
언어별 Catalog Tribe 개발자 비율
(2018)
17%
Kotlin
40%
Scala
13%
Java
30%
Java Scala Kotlin Python

66. Resources
• Kotlin Resources
• kotlinx.coroutines
• 20 Excellent Resources for learning Kotlin
• Books
• Kotlin in action, Joy of Kotlin
• Try Kotlin

67. Q&A

68. Thank you!